Positioning tracking method, device, equipment, medium and program

Abstract

The embodiment of the invention provides a positioning tracking method and device, equipment, a medium and a program, camera modules are arranged on auxiliary positioning devices, the auxiliary positioning devices realize self-tracking based on the camera modules and IMU sensors of the auxiliary positioning devices, and the positioning results of the auxiliary positioning devices are sent to a head-mounted display; the head-mounted display is further provided with an anchor point emitter, the camera of the head-mounted display and the camera of the auxiliary positioning device shoot light spots emitted by the anchor point emitter to obtain anchor point images, and the head-mounted display achieves alignment of a coordinate system of the auxiliary positioning device and a coordinate system of the head-mounted display based on the anchor point images. And positioning the auxiliary positioning device according to the aligned coordinate system. Since each auxiliary positioning device performs positioning based on the image shot by the own camera and the IMU data, the positioning is not limited by the field angle of the head-mounted display, coordinate system alignment is realized through the anchor point image, the tracking error is reduced, and the tracking precision is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of virtual reality technology, and in particular to a positioning tracking method, device, setting, medium and program. Background Art

[0002] At present, the application scenarios of extended reality (XR) technology are becoming more and more extensive. XR technology specifically includes virtual reality (VR) technology, augmented reality (AR) technology and mixed reality (MR) technology. XR technology enables users to immersively experience various virtual scenes.

[0003] In XR game scenes, human motion capture technology is used to detect the posture information and motion trajectory of the human body in three-dimensional space, and simulate human motion in virtual scenes. Human motion capture is also called human positioning tracking. In the prior art, the camera in the head-mounted display of the integrated XR device is usually used for positioning. The camera continuously collects various environmental images during the movement process to determine the position information of the head-mounted display. At the same time, optical tracking is used to determine the relative position relationship between other interactive devices of XR (handles or trackers) and the head-mounted display, so as to determine the position information of each other interactive device.

[0004] However, optical tracking is limited by the field of view (FOV) of each camera in the head-mounted display, and can only track the relative position relationship between other interactive devices within the FOV and the head-mounted display. When an interactive device is outside the FOV, it cannot be positioned and tracked, which makes the positioning and tracking of all-in-one VR smart devices have certain limitations. Summary of the invention

[0005] The embodiments of the present application provide a positioning and tracking method, apparatus, device, medium and program. The auxiliary positioning device of the XR device performs positioning based on the images captured by its own camera and IMU data, which is limited to the field of view of the head-mounted display, and realizes coordinate system alignment through anchor point images, thereby reducing tracking errors and improving tracking accuracy.

[0006] In a first aspect, an embodiment of the present application provides a positioning and tracking method, which is applied to an extended reality XR device, wherein the XR device includes a head-mounted display and an auxiliary positioning device, wherein the head-mounted display includes a first camera module, a first inertial measurement unit IMU, and an anchor point transmitter, wherein the auxiliary positioning device includes a second camera module and a second IMU, and the anchor point transmitter is used to emit a light spot; the method includes:

[0007] The head mounted display obtains a first positioning result of the head mounted display according to a first tracking image acquired by the first camera module and first IMU data measured by the first IMU;

[0008] The auxiliary positioning device obtains a second positioning result of the auxiliary positioning device according to the second tracking image collected by the second camera module and the second IMU data measured by the second IMU, and sends the second positioning result to the head mounted display;

[0009] The first camera acquires a first anchor point image, and the second camera acquires a second anchor point image corresponding to the first anchor point image, wherein the first anchor point image and the second anchor point image include images of the light spot emitted by the anchor point emitter;

[0010] The head mounted display determines, according to the first anchor point image and the second anchor point image, a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device;

[0011] According to the conversion relationship and the second positioning result, the position and posture information of the auxiliary positioning device in the coordinate system of the head mounted display is determined.

[0012] In a second aspect, an embodiment of the present application provides a positioning and tracking method, which is applied to a head-mounted display of an XR device, wherein the XR device further includes an auxiliary positioning device, the head-mounted display includes a first camera module, a first inertial measurement unit IMU, and an anchor point transmitter, the auxiliary positioning device includes a second camera module and a second IMU, and the anchor point transmitter is used to emit a light spot; the method includes:

[0013] Acquire a first positioning result of the head mounted display according to a first tracking image acquired by the first camera module and first IMU data measured by the first IMU;

[0014] receiving a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, where the second positioning result is determined by the auxiliary positioning device according to a second tracking image collected by the second camera and second IMU data measured by the second IMU;

[0015] Acquire a first anchor point image captured by the first camera and a second anchor point image captured by the second camera, wherein the first anchor point image and the second anchor point image include images of light spots emitted by the anchor point emitter;

[0016] Determining a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image;

[0017] According to the conversion relationship and the second positioning result, the position and posture information of the auxiliary positioning device in the coordinate system of the head mounted display is determined.

[0018] In a third aspect, an embodiment of the present application provides a positioning tracking device, which is configured in a head mounted display of the XR device described in the second aspect, and the device includes:

[0019] A positioning module, configured to obtain a first positioning result of the head mounted display according to a first tracking image acquired by a first camera module and first IMU data obtained by first IMU measurement;

[0020] A receiving module, configured to receive a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, wherein the second positioning result is determined by the auxiliary positioning device according to a second tracking image collected by a second camera and second IMU data measured by a second IMU;

[0021] an acquisition module, configured to acquire a first anchor point image acquired by the first camera and a second anchor point image acquired by the second camera, wherein the first anchor point image and the second anchor point image include images of light spots emitted by the anchor point emitter;

[0022] a determination module, configured to determine a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image;

[0023] A coordinate conversion module is used to determine the position information of the auxiliary positioning device in the coordinate system of the head mounted display according to the conversion relationship and the second positioning result.

[0024] In a fourth aspect, an embodiment of the present application provides an XR device, wherein the XR device includes:

[0025] A head mounted display, comprising a first camera module, a first inertial measurement unit IMU and an anchor point transmitter, wherein the anchor point transmitter is used to emit a light spot;

[0026] an auxiliary positioning device, comprising a handle and a body tracker worn on different body parts, the auxiliary positioning device comprising a second camera module and a second IMU;

[0027] The head mounted display and the auxiliary positioning device are used to execute the method described in the first aspect of the present application.

[0028] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute the method described in the second aspect above.

[0029] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the method described in the second aspect above is implemented.

[0030] The positioning and tracking method, device, equipment, medium and program provided in the embodiments of the present application, by setting a camera module on the auxiliary positioning device of the XR device, each auxiliary positioning device can realize self-tracking based on its own camera module and IMU sensor, and send its own positioning result to the head-mounted display. In addition, the head-mounted display is also equipped with an anchor point transmitter. The cameras of the head-mounted display and the auxiliary positioning device shoot the light spot emitted by the anchor point transmitter to obtain an anchor point image. The head-mounted display realizes the alignment of the coordinate system of the auxiliary positioning device and the coordinate system of the head-mounted display based on the anchor point image, and determines the position information of the auxiliary positioning device in the coordinate system of the head-mounted display according to the aligned coordinate system. Since each auxiliary positioning device is positioned based on the image captured by its own camera and the IMU data, it does not need to be limited by the field of view of the head-mounted display, and the coordinate system alignment is realized through the anchor point image, which reduces the tracking error and improves the tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 It is a structural schematic diagram of an XR device in the prior art;

[0033] Figure 2 This is a structural block diagram of an XR device shown in an embodiment of the present application;

[0034] Figure 3 Schematic diagram of the light spot emitted by the anchor transmitter;

[0035] Figure 4 A flowchart of the positioning and tracking method provided in Example 1 of the present application;

[0036] Figure 5 This is a system time diagram of the head mounted display and auxiliary positioning device;

[0037] Figure 6 A flowchart of a synchronization method in a positioning and tracking process provided in Embodiment 2 of the present application;

[0038] Figure 7 A schematic diagram of synchronization between the wireless chip and the processor of the head mounted display;

[0039] Figure 8 It is a schematic diagram of synchronization between a head mounted display and an auxiliary positioning device;

[0040] Fig. 9 A flowchart of the positioning and tracking method provided in Embodiment 3 of the present application;

[0041] Fig.10 A schematic diagram of software functional modules when the head mounted display and auxiliary positioning device are positioned using the SLAM method;

[0042] Fig.11 A flowchart of a positioning and tracking method provided in Embodiment 4 of the present application;

[0043] Fig.12 A schematic diagram of the structure of a positioning and tracking device provided in Example 5 of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0046] The embodiment of the present application provides a positioning and tracking method, which can be applied to an XR device, wherein the XR device includes a head-mounted display and an auxiliary positioning device, also called a positioning device or an auxiliary positioning device, wherein the auxiliary positioning device includes a handle and a body tracker worn on different body parts. The number of handles is usually two, worn on the left and right hands of the user, and the number of body trackers can be two or more, worn on the user's wrist, calf, thigh, waist, etc. The body trackers worn on different body parts are used to track the motion data of different body parts.

[0047] Existing XR devices usually use optical tracking to track head-mounted displays and auxiliary positioning devices. Figure 1 It is a schematic diagram of the structure of an XR device in the prior art, and various functional modules configured in the head-mounted display and the auxiliary positioning device of the XR device are as follows: Figure 1 shown.

[0048] The positioning principle of existing XR devices includes: the head-mounted display collects optical images of the environment through the configured camera, and the left and right handles are equipped with light-emitting diode modules to facilitate the camera module in the head-mounted display to collect images. In addition, the head-mounted display, the left and right handles, and each body tracker are equipped with inertial measurement unit (IMU) sensors to assist in positioning tracking based on optical tracking.

[0049] Specifically, the head-mounted display can collect optical images of the environment in real time through the camera module, and realize the positioning and tracking of the head-mounted display based on the optical image combined with the relevant position information detected by the IMU sensor on the head-mounted display. Then, based on the position of the auxiliary positioning device in the optical image, the relative position relationship between the auxiliary positioning device and the head-mounted display is analyzed, and combined with the relevant position information detected by the IMU sensors on the left and right handles and each body tracker, the positioning and tracking of the auxiliary positioning devices such as the left and right handles and each body tracker is realized.

[0050] As mentioned above, when the head-mounted display and auxiliary positioning devices of the XR device are positioned and tracked by optical tracking, it will be limited by the field of view of the head-mounted display. When a certain auxiliary positioning device is outside the field of view, the collected optical image cannot represent the movement position of the auxiliary positioning device, and the auxiliary positioning device cannot be positioned and tracked, which makes the positioning tracking of the XR device have certain limitations.

[0051] In order to solve the above problems, this application adds camera modules and processors to each auxiliary positioning device based on the existing structure of the XR device. Each auxiliary positioning device can achieve self-tracking based on its own camera module and IMU sensor, and send its own positioning information to the head-mounted display. In addition, the head-mounted display is also equipped with an anchor transmitter, which is used to emit light spots. The cameras of the head-mounted display and the auxiliary positioning device shoot the light spots emitted by the anchor transmitter at the same time to obtain an anchor image. The head-mounted display aligns the coordinate system of the auxiliary positioning device with the coordinate system of the head-mounted display based on the anchor image, and positions the auxiliary positioning device according to the aligned coordinate system. In this positioning method, the auxiliary positioning device is positioned based on the image captured by its own camera and the IMU data, without being limited by the field of view of the head-mounted display, and the coordinate system is aligned through the anchor image, which reduces the tracking error and improves the tracking accuracy.

[0052] Figure 2 FIG. 1 is a structural block diagram of an XR device shown in an embodiment of the present application. Figure 2 As shown, the XR includes a head mounted display, a left handle, a right handle and two body trackers, wherein the left handle and the right handle have the same structure, and the two body trackers have the same structure. The embodiment of the present application only describes the functions of the modules involved in the positioning and tracking method provided by the present application, and the functions of other modules are not described in detail.

[0053] The head-mounted display includes a camera module, an IMU sensor, a processor and an anchor transmitter, wherein the processor is used to realize the 6-degree-of-freedom (Dof) positioning of the head-mounted display according to the tracking image collected by the camera module and the IMU data collected by the IMU sensor, and the 6Dof positioning data includes the position information and posture information (referred to as posture information) of the head-mounted display.

[0054] The anchor point transmitter is used to emit light spots, and the number of light spots can be one or more. The camera module of the head-mounted display and the camera module of the auxiliary positioning device capture the light spots emitted by the anchor point transmitter to obtain an anchor point image, which includes the light spots. The light spots in the anchor point image are called anchor points. The head-mounted display is based on the anchor point image captured by its own camera module and the anchor point image captured by the auxiliary positioning device, so as to align the coordinate system of the auxiliary positioning device with the coordinate system of the head-mounted display.

[0055] The anchor emitter can be a structured light sensor or a time of flight (ToF) sensor, wherein structured light is a system structure composed of a projector and a camera. After the projector projects specific light information onto the surface and background of an object, the camera collects it and calculates the position and depth of the object according to the change of the light signal caused by the object, thereby restoring the entire three-dimensional space. In this embodiment, the anchor emitter is a structured light projector, and the camera module of the head-mounted display and the camera module of the auxiliary positioning device and the anchor emitter form a structured light. The ToF sensor can emit infrared light to the surface of an object and shoot the infrared light through the camera.

[0056] The handle includes at least a camera module, an IMU sensor and a processor, wherein the processor is used to realize 6Dof tracking of the handle according to the tracking image collected by the camera module and the IMU data collected by the IMU sensor.

[0057] The body tracker includes at least a camera module, an IMU sensor and a processor, wherein the processor is used to implement 6Dof tracking of the body tracker based on the tracking image collected by the camera module and the IMU data collected by the IMU sensor.

[0058] Figure 2 The head mounted display, handle and body tracker also include a wireless communication module, and the handle and body tracker realize wireless communication with the head mounted display through the wireless communication module, and the wireless communication can adopt a communication protocol such as a 2.4G network transmission protocol or a low power consumption Bluetooth protocol. It is understood that the handle and body tracker can also communicate with the head mounted display through a wired manner, and the implementation of this application is not limited to this.

[0059] In the embodiment of the present application, the images captured by the camera modules of the head-mounted display, the handle and the body tracker include two types of images: tracking images and anchor point images. The tracking images are used to track the head-mounted display, the handle and the body tracker, and the tracking images capture images of the real environment in which the head-mounted display, the handle and the body tracker are located.

[0060] An anchor image refers to an image captured based on the light spot emitted by the anchor transmitter. The light spot emitted by the anchor transmitter can be visible light or invisible light. For example, the anchor transmitter uses a laser to emit multiple light spots. The user cannot see the light spots, but the camera can capture the light spots.

[0061] Figure 3 This is a schematic diagram of the light spot emitted by the anchor emitter, such as Figure 3As shown, the anchor point transmitter emits multiple light spots, and the camera modules of the head-mounted display, the handle and the body tracker respectively shoot the light spots. The camera modules of the head-mounted display, the handle and the body tracker are respectively located at different positions and angles to shoot the light spots. The head-mounted display can solve the relative position relationship between the anchor point and the camera based on the anchor point images shot at different angles and positions. According to the position relationship between the same anchor point and different cameras, the conversion relationship between the coordinate system of the head-mounted display and the body tracker, as well as the conversion relationship between the coordinate system of the head-mounted display and the handle can be obtained. Among them, the relative position relationship between the anchor point and the camera can also be understood as the relative position relationship between the anchor point and the device where the camera is located.

[0062] Figure 4 This is a flowchart of a positioning tracking method provided in Embodiment 1 of the present application. The method of this embodiment comprises Figure 2 The XR device shown in the figure is executed. In this embodiment, for the sake of clarity, the camera module and IMU on the head-mounted display are referred to as the first camera module and the first IMU, the camera module and IMU on the auxiliary positioning device are referred to as the second camera module and the second IMU, the tracking image and the anchor image captured by the head-mounted display are referred to as the first tracking image and the first anchor image, and the tracking image and the anchor image captured by the auxiliary positioning device are referred to as the second tracking image and the second anchor image. Figure 4 As shown, the positioning tracking method includes the following steps.

[0063] S101: The head mounted display obtains a first positioning result of the head mounted display according to a first tracking image acquired by a first camera module and first IMU data obtained by first IMU measurement.

[0064] The first positioning result includes the 6Dof positioning data of the head-mounted display. The 6Dof positioning data includes position movement information (3 degrees of freedom) on the X, Y, and Z axes and rotation information on the X, Y, and Z axes. Therefore, it can not only detect the changes in the field of view angle caused by the rotation of body parts, but also detect the changes in the field of view caused by the movement of body parts.

[0065] In this embodiment, the head-mounted display can be positioned using a simultaneous localization and mapping (SLAM) positioning method. The SLAM positioning method determines the current position information of the head-mounted display by analyzing the position of each environmental feature point in the tracking image relative to the head-mounted display, and obtains the rotation information of the head-mounted display based on the first IMU data measured by the first IMU, thereby acquiring 6Dof tracking data.

[0066] S102: The auxiliary positioning device obtains a second positioning result of the auxiliary positioning device according to the second tracking image collected by the second camera module and the second IMU data obtained by the second IMU measurement, and sends the second positioning result to the head mounted display.

[0067] By setting a camera on the auxiliary positioning device, the auxiliary positioning device can also use the SLAM positioning method for positioning. SLAM positioning is a self-tracking method, that is, the device can achieve positioning by relying on its own components, without the need for other devices to assist in positioning. The second positioning result obtained by the auxiliary positioning device is the 6Dof positioning data of the auxiliary positioning device.

[0068] It should be noted that the handle can be positioned not only by the SLAM positioning method, but also by other methods, such as infrared positioning, by emitting a light spot through the light unit on the handle, and the camera of the head-mounted display captures the light spot emitted by the light unit for positioning.

[0069] S103: The first camera acquires a first anchor point image, and the second camera acquires a second anchor point image corresponding to the first anchor point image, wherein the first anchor point image and the second anchor point image include images of light spots emitted by the anchor point transmitter.

[0070] The anchor point transmitter can periodically emit light spots, and the head-mounted display and the auxiliary positioning device collect anchor point images at a fixed time. The first anchor point image and the second anchor point image are anchor point images taken at the same time. The head-mounted display determines the conversion relationship between the coordinate systems of the head-mounted display and the auxiliary positioning device based on the anchor point images taken at the same time. Usually, the acquisition frequency of the anchor point image is lower than the acquisition frequency of the tracking image. Usually, 30 or 60 frames of tracking images are collected in one second, while the anchor point image can be collected once a second, or once a minute, or once more.

[0071] The first anchor point image and the second anchor point image used to determine the conversion relationship are images of the light spot taken at the same time. In order to improve the accuracy of the coordinate system conversion, the first camera module and each second camera module synchronously shoot the anchor point image, that is, the first camera module and the second camera module are time-synchronized when shooting images to ensure that the first anchor point image and the second anchor image are images shot at the same time.

[0072] Optionally, the exposure time of the first camera module and the second camera module for capturing the anchor image is shorter than the exposure time for capturing the tracking image. By reducing the exposure time of the camera when capturing the anchor image, the impact of the environment on the anchor image can be reduced.

[0073] Since the internal parameters of each camera module are different, the first camera module and the second camera module use different exposure times when capturing the anchor point image.

[0074] S104: The head mounted display determines a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image.

[0075] In order to reduce the power consumption of the device, the head mounted display can periodically determine the conversion relationship between the coordinate system of the head mounted display and the coordinate system of the auxiliary positioning device according to the anchor point image, so as to align the coordinate system of the head mounted display with the coordinate system of the auxiliary positioning device. The conversion relationship between the coordinate system of the head mounted display and the coordinate system of the auxiliary positioning device can be a conversion matrix.

[0076] The head-mounted display and the auxiliary positioning device each have their own coordinate systems. Therefore, it is necessary to unify the positioning data of the auxiliary positioning device into the coordinate system of the head-mounted display, that is, to convert the positioning data of the auxiliary positioning device from the coordinate system of the auxiliary positioning device to the coordinate system of the head-mounted display. The conversion of the coordinate system is also called coordinate system alignment.

[0077] S105: Determine the position and posture information of the auxiliary positioning device in the coordinate system of the head mounted display according to the conversion relationship and the second positioning result.

[0078] The conversion relationship between the coordinate system of the head-mounted display and different auxiliary positioning devices is different. After determining the conversion relationship between the coordinate system of the head-mounted display and different auxiliary positioning devices, the head-mounted display saves the conversion relationship between the coordinate system of the head-mounted display and different auxiliary positioning devices. When the head-mounted display obtains the positioning result based on each frame of the tracking image, the positioning result of the auxiliary positioning device is converted into a coordinate system based on the conversion relationship to obtain the positioning result of the auxiliary positioning device in the coordinate system of the head-mounted display. The positioning result of the auxiliary positioning device in the coordinate system of the head-mounted display is the posture information of the auxiliary positioning device in the coordinate system of the head-mounted display.

[0079] After converting the second positioning result of the auxiliary positioning device to the coordinate system of the head-mounted display, the positioning results of the head-mounted display, the handle and the body tracker in the coordinate system of the head-mounted display are obtained. According to the positioning results of the body tracker, the handle and the head-mounted display, the corresponding user motion posture can be identified, and the motion posture can include user movements and postures. This tracking method is no longer limited to the field of view of the head-mounted display in the XR device, and can accurately track the body tracker and the handle.

[0080] In this embodiment, a camera module and a processor are added to the auxiliary positioning device of the XR device. The auxiliary positioning device can realize self-tracking based on its own camera module and IMU sensor, and send its own positioning results to the head-mounted display. In addition, the head-mounted display is also equipped with an anchor transmitter. The cameras of the head-mounted display and the auxiliary positioning device shoot the light spot emitted by the anchor transmitter to obtain an anchor image. The head-mounted display aligns the coordinate system of the auxiliary positioning device with the coordinate system of the head-mounted display based on the anchor image, and locates the auxiliary positioning device according to the aligned coordinate system. Since each auxiliary positioning device is located based on the image captured by its own camera and IMU data, it does not need to be limited by the field of view of the head-mounted display, and the coordinate system is aligned through the anchor image, which reduces the tracking error and improves the tracking accuracy.

[0081] On the basis of the first embodiment, in order to improve the tracking accuracy, it is necessary to keep the entire device synchronized. On the basis of device synchronization, multiple camera modules of the head mounted display and the auxiliary positioning device are controlled to simultaneously capture anchor point images. Figure 5 This is a system time diagram of the head mounted display and auxiliary positioning device, such as Figure 5 As shown, the processor, wireless chip, left handle, right handle, body tracker 1 and body tracker 2 of the head-mounted display each use independent system time, and the system time of each device is not synchronized. The ultimate goal of synchronization is: the system time T of the processor of the head-mounted display, the system time T1 of the wireless chip of the head-mounted display, the system time T2 of the left handle, the system time T3 of the right handle, the system time T4 of the body tracker 1 and the system time T5 of the body tracker 2 are completely synchronized, and the synchronization error is required to be within 5us.

[0082] Figure 6 This is a flowchart of the synchronization method in the positioning tracking process provided in Example 2 of the present application, referring to Figure 5 and Figure 6 , the method provided in this embodiment comprises the following steps:

[0083] S201. The wireless chip and processor of the head mounted display achieve time synchronization through interrupt operation and a second synchronization signal.

[0084] The head mounted display includes a wireless chip and a processor, that is, the time T1 of the wireless chip is synchronized to the time T of the processor. Figure 7 A schematic diagram of the synchronization of the wireless chip and processor of the head mounted display, such as Figure 7As shown, the synchronization between the wireless chip and the processor is achieved through interruption and data communication. Specifically, the wireless chip reads the local time t1 and triggers an interrupt through hardware. The processor triggers the internal timing t during the interrupt. The wireless chip sends t1 to the processor through the communication interface. The signal carrying t1 is the second synchronization signal. When the processor receives t1, it compares t1 and t to obtain the difference △t1 between the two. The time of the wireless chip can be converted into the time of the processor through △t1.

[0085] S202: The head mounted display sends a first synchronization signal to the auxiliary positioning device, and the auxiliary positioning device achieves time synchronization with the head mounted display according to the first synchronization signal.

[0086] Step S201 realizes the synchronization between T1 and T, and this step realizes the synchronization between T1 and T2, T3, T4, and T5. Figure 8 It is a schematic diagram of synchronization between a head mounted display and an auxiliary positioning device, such as Figure 8 As shown, the head-mounted display has a global timer: global timer 1. When the application is performing the synchronization function, after the data is ready to be sent, the global timer 1 automatically fills the local time t1 into the first synchronization signal in the data sending buffer, and the wireless modulation module of the head-mounted display sends the first synchronization signal carrying t1 to the auxiliary positioning device wirelessly.

[0087] After the auxiliary positioning device is wirelessly demodulated, the global timer 2 of the auxiliary positioning device automatically fills the local time t2 into the data buffer of the application. t2 is the reception time of the first synchronization signal. The application of the auxiliary positioning device obtains the difference △t2 between t1 and t2 based on t1 and t2. The time of the auxiliary positioning device and the time of the head-mounted display can be synchronized through △t2.

[0088] It should be noted that the first synchronization signal of the head-mounted display and the auxiliary positioning device has no retransmission mechanism, that is, in the current synchronization cycle, if the transmission of the first synchronization signal fails, the first synchronization signal will not be retransmitted, but will wait for the next synchronization cycle, thereby ensuring that the time interval △t2 between wireless modulation and wireless demodulation is a fixed value.

[0089] S203: The head mounted display synchronizes the multiple cameras included in the first camera module.

[0090] Steps S201 and S202 realize the time synchronization between the processor and the wireless chip of the head-mounted display, and the system time synchronization between the head-mounted display and other auxiliary positioning devices. Based on the system time synchronization of the devices, the cameras of the head-mounted display and the auxiliary positioning devices are controlled to synchronously capture the anchor point images.

[0091] When the head-mounted display includes multiple cameras, it is necessary to ensure that the multiple cameras work synchronously, that is, to synchronously capture tracking images and anchor point images. Due to the different positions of each camera, the external environment corresponding to each camera will be different. In order to ensure the tracking accuracy and the consistency of each optical image output in different environments, the exposure parameters of each camera are also different. For example, the exposure setting of the camera with a darker environment is longer, and the exposure setting of the camera with a better environment is shorter. In this case, the exposure center points of the optical images captured by each camera can be aligned, thereby ensuring that the multiple cameras are consistent in the time when capturing images. Multiple cameras can capture tracking images and anchor point images at the same time, ensuring the tracking accuracy and coordinate system alignment accuracy.

[0092] S204: The head mounted display determines the shooting time information of the anchor point image according to the emission time information of the anchor point transmitter.

[0093] The emission time information of the anchor transmitter can be the emission frequency of the anchor transmitter. The emission frequency indicates the number of emission times of the light spot per unit time. The greater the emission frequency, the greater the number of emission times and the smaller the emission interval. The emission time information of the anchor transmitter can also be the emission interval and the start emission time of the anchor transmitter.

[0094] The anchor point transmitter emits a light spot at a fixed time according to the emission time information. Since the emission time of the light spot emitted by the anchor point transmitter is fixed, the shooting time information of the anchor point image can be determined according to the emission time of the anchor point transmitter, and the camera can be controlled to shoot the anchor point image at a fixed time.

[0095] The shooting time information of the anchor image can be the acquisition frequency of the anchor image. The acquisition frequency can be understood as the number of acquisitions per unit time. The higher the acquisition frequency per unit time, the shorter the acquisition interval, that is, the shorter the acquisition interval of the anchor image by the camera module. The shooting time information of the anchor image can also be the acquisition interval and the acquisition start time.

[0096] The head mounted display determines the shooting time information of the anchor image according to the emission time information of the anchor transmitter, and can ensure that the light spot emitted by the anchor transmitter is captured by the camera.

[0097] Optionally, the shooting time information of the anchor image is the same as the emission time information of the anchor transmitter, and accordingly, the emission time of the anchor transmitter is the same as the acquisition time of the anchor image, or the acquisition time of the anchor image overlaps and is greater than the emission time of the anchor transmitter, thereby ensuring that the camera can capture the anchor image.

[0098] In an optional implementation, the anchor point transmitter can emit a light spot using a fixed emission frequency, which is pre-configured. The emission frequency can be set according to the tracking accuracy. If the device or application environment has high requirements for tracking accuracy, the emission frequency can be set higher. If the device or application environment has low requirements for tracking accuracy, the emission frequency can be set lower. While meeting the tracking accuracy, the energy consumption of coordinate system alignment also needs to be considered.

[0099] In another optional implementation, the anchor point transmitter uses a first transmission frequency within a first period of time after the head mounted display is turned on, and uses a second transmission frequency after the first period of time, and the first transmission frequency is greater than the second transmission frequency.

[0100] This is because, when the head-mounted display and the auxiliary positioning device are just turned on, the head-mounted display and the auxiliary positioning device perform self-tracking respectively, and each uses a point cloud database composed of the feature point data of the environment it obtains. At this time, the point cloud databases used by the two are not synchronized, and the point cloud databases are incomplete and inaccurate. The point cloud database cannot be used for coordinate system alignment, and only the anchor point image can be used for coordinate system alignment. Therefore, a larger transmission frequency can be used for coordinate system alignment to reduce tracking errors and ensure tracking accuracy. Subsequently, as the equipment runs, each point cloud database is gradually improved, and the head-mounted display can also share its own point cloud database with the auxiliary positioning device to achieve consistency between the two point cloud databases. At this time, the point cloud database can be used for coordinate alignment. Therefore, a smaller transmission frequency can be used for coordinate system alignment.

[0101] S205: The head mounted display sends the shooting time information of the anchor point image to the auxiliary positioning device.

[0102] The shooting time information of the anchor point image used by the head mounted display is the same as the shooting time information of the anchor point image used by the auxiliary positioning device, thereby ensuring that the head mounted display and each auxiliary positioning device shoot the anchor point image at the same time.

[0103] S206. The head mounted display controls the first camera module to capture the first anchor point image according to the shooting time information of the anchor point image, and the auxiliary positioning device controls the second camera module to capture the second anchor point image according to the shooting time information of the anchor point image.

[0104] The head-mounted display and the auxiliary positioning device are shot based on the same anchor image shooting time information, so that the shooting time of the first anchor image and the second anchor image is the same. The coordinate system is aligned based on the time-synchronized first anchor image and the second anchor image, which can improve the tracking accuracy.

[0105] In this embodiment, after the head mounted display performs system time synchronization with the auxiliary positioning device, the head mounted display determines the shooting time information of the anchor image according to the emission time information of the anchor transmitter, and sends the shooting time information of the anchor image to the auxiliary positioning device; the head mounted display controls the first camera module to shoot the first anchor image according to the shooting time information of the anchor image; the auxiliary positioning device controls the second camera module to shoot the second anchor image according to the shooting time information of the anchor image. The shooting time information of the anchor image used by the head mounted display is the same as the shooting time information of the anchor image used by the auxiliary positioning device, so that the head mounted display and the camera of the auxiliary positioning device synchronously shoot the anchor image, and the coordinate system is aligned based on the synchronously shot anchor image, thereby improving the tracking accuracy.

[0106] Based on Embodiment 1 and Embodiment 2, Embodiment 3 of the present application provides a positioning and tracking method for explaining step S104 in Embodiment 1, namely, determining the conversion relationship between the coordinate systems of the head-mounted display and the auxiliary positioning device based on the first anchor point image captured by the first camera and the second anchor point image captured by the second camera. Fig. 9 This is a flowchart of the positioning tracking method provided in Example 3 of the present application, such as Fig. 9 As shown, the method provided in this embodiment includes the following steps:

[0107] S1041: The head mounted display determines first relative position information of the anchor point relative to the head mounted display according to the first anchor point image.

[0108] The anchor point transmitter emits multiple light spots at a fixed time according to the configured emission time information. The first camera module captures the first anchor point image according to the shooting time information of the anchor point image. The second camera module captures the second anchor point image according to the shooting time information of the anchor point image.

[0109] The first relative position information is the 3D coordinates of the anchor point in the coordinate system of the head-mounted display. The first anchor point image includes multiple anchor points. The head-mounted display determines the 3D coordinates of each anchor point image based on the first anchor point image. The 3D coordinates of the anchor point determined by the head-mounted display based on the first anchor point image are the 3D coordinates of the anchor point in the coordinate system of the head-mounted display, not the absolute coordinates of the anchor point in the world coordinate system. Therefore, the 3D coordinates of the anchor point determined by the head-mounted display are the relative position of the anchor point relative to the head-mounted display.

[0110] S1042: The auxiliary positioning device determines second relative position information of the anchor point relative to the auxiliary positioning device based on the second anchor point image, and sends the second relative position information to the head mounted display.

[0111] The second relative position information is the 3D coordinates of the anchor point in the auxiliary positioning device coordinate system. The auxiliary positioning device obtains the 3D coordinates of the anchor point in the auxiliary positioning device coordinate system using the same method as the head mounted display.

[0112] S1043: The head mounted display determines a conversion relationship between the coordinate systems of the head mounted display and the auxiliary positioning device according to the second relative position information and the first relative position information of the anchor point sent by the auxiliary positioning device.

[0113] It can be understood that when the head-mounted display determines the conversion relationship between the head-mounted display and the auxiliary positioning device, it needs to use the first relative position information and the second relative position information of the same anchor point. Due to the different positions of the head-mounted display and the auxiliary positioning device, the number of anchor points photographed may be the same or different. For example, the anchor point transmitter emits 10 light spots, the head-mounted display detects 10 anchor points, and the auxiliary positioning device detects 8 anchor points. In this case, only all or part of the 8 anchor points detected by the auxiliary positioning device can be used to determine the conversion relationship.

[0114] When there are multiple auxiliary positioning devices, the head-mounted display will determine the conversion relationship between itself and the coordinate system of each auxiliary positioning device.

[0115] Optionally, in other embodiments of the present application, the auxiliary positioning device may not detect the second anchor point information, but instead send the second anchor point image to the head-mounted display, and the head-mounted display determines the second relative position information of the anchor point relative to the auxiliary positioning device based on the second anchor point image, and determines the first relative position information of the anchor point relative to the head-mounted display based on the first anchor point image, and determines the transformation matrix between the coordinate system of the auxiliary positioning device and the coordinate system of the head-mounted display based on the first relative position and the second relative position of the anchor point.

[0116] In this embodiment, the head mounted display, handle and body tracker can all be positioned using the SLAM positioning method. Fig.10 The schematic diagram of software functional modules for head mounted display and auxiliary positioning device using SLAM method for positioning is shown as follows: Fig.10 As shown,

[0117] The point cloud data module of the head mounted display is used to store or maintain the feature point data of the environment in which the head mounted display is located, that is, the feature point data of the environment in which the head mounted display is performing 6Dof tracking. The feature point data of the environment is also called point cloud data. The feature point data includes the 3D coordinates of the feature points, and may also include the depth information of the feature points, the descriptors of the feature points, etc. The feature point data of the environment in which the head mounted display is located may be stored in the first point cloud database in the point cloud data module. The point cloud data module of the head mounted display may determine the position of the environment in which the head mounted display is located through the feature point data.

[0118] During the use of the head-mounted display, the point cloud data update module of the head-mounted display continuously collects more environmental feature points, updates the point cloud data of the environment in real time, and updates the local point cloud data module. The head-mounted display handle point cloud wireless sharing module mainly includes two basic functions: the first function is to update the local point cloud data module to the auxiliary positioning device during the power-on and non-use stage, so that the auxiliary positioning device can track the position based on the same point cloud data and the same world coordinate system as the head-mounted display; the second function is to complete the point cloud data update of the two systems through this module after the point cloud data of the head-mounted display and the auxiliary positioning device are updated.

[0119] The point cloud database of the head mounted display includes multiple different point cloud databases, and different point cloud databases correspond to different environments. For example, the point cloud database includes three point cloud databases, namely the point cloud databases of room 1, room 2 and room 3. After the head mounted display is turned on, the head mounted tracking module will collect the feature point data of the current environment, the point cloud database selection module will match the feature point data of the current environment with the point cloud database data, and update the successfully matched point cloud database to the point cloud data module. After the local point cloud data is updated, the point cloud data update module will also update the point cloud database.

[0120] The head-mounted tracking module combines the data provided by the head-mounted display's camera module, IMU sensor module, and point cloud data module to complete the position tracking of the head-mounted display in the current environment.

[0121] The functions of the modules of the auxiliary positioning device are similar to those of the head mounted display, and will not be described again here. Compared with the head mounted display, the auxiliary positioning device shown in the figure lacks the point cloud database, point cloud database selection and anchor point emission modules.

[0122] The head mounted display obtains a first positioning result according to the first tracking image, the first IMU data, and a first point cloud database of the head mounted display, wherein the first point cloud database includes feature point data of the environment in which the head mounted display is located. The auxiliary positioning device obtains a second positioning result according to the second tracking image, the second IMU data, and a second point cloud database of the auxiliary positioning device, wherein the second point cloud database includes feature point data of the environment in which the auxiliary positioning device is located.

[0123] In one implementation, after the head-mounted display and the auxiliary positioning device are turned on, each determines its own point cloud database through tracking images, and continuously updates and improves its own point cloud database.

[0124] In another implementation, before the head-mounted display is turned on and operated, if there is no matching first point cloud database in the point cloud database of the head-mounted display, the head-mounted display and the auxiliary positioning device each determine their own point cloud database through tracking images, and continuously update and improve their own point cloud databases.

[0125] Optionally, before the head-mounted display is turned on and not running, if there is a matching first point cloud database in the point cloud database of the head-mounted display, the head-mounted display can send the first point cloud database to the auxiliary positioning device. Subsequently, the head-mounted display updates the first point cloud database according to the tracking image collected by the first camera module, and the auxiliary positioning device updates the second point cloud database according to the tracking image collected by the second camera module, that is, the head-mounted display synchronizes the point cloud database to the auxiliary positioning device after it is turned on, and subsequently, the head-mounted display and the auxiliary positioning device each update their own point cloud database.

[0126] Optionally, the auxiliary positioning device can also send the updated data of the second point cloud database to the head mounted display, and the head mounted display updates the first point cloud database according to the updated data. The first point cloud database maintained on the head mounted display is the most complete point cloud database. In order to improve the tracking accuracy, the head mounted display can also regularly share the first point cloud database with the auxiliary positioning device.

[0127] The above method embodiment describes the positioning tracking method of the XR device from the perspective of the interaction between the head-mounted display and each auxiliary positioning device. Embodiment 4 of the present application describes the positioning tracking method from the perspective of the head-mounted display. Fig.11 This is a flowchart of a positioning tracking method provided in Embodiment 4 of the present application. The method of this embodiment is executed by the head mounted display shown above. Fig.11 As shown, the method provided in this embodiment includes the following steps.

[0128] S301. Obtain a first positioning result of a head mounted display according to a first tracking image acquired by a first camera module and first IMU data obtained by first IMU measurement.

[0129] S302, receiving a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, where the second positioning result is determined by the auxiliary positioning device according to a second tracking image captured by a second camera and second IMU data measured by a second IMU.

[0130] S303: Acquire a first anchor point image captured by the first camera and a second anchor point image captured by the second camera, wherein the first anchor point image and the second anchor point image include images of light spots emitted by the anchor point emitter.

[0131] S304: Determine a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image.

[0132] S305: Determine the position and posture information of the auxiliary positioning device in the coordinate system of the head mounted display according to the conversion relationship and the second positioning result of the auxiliary positioning device.

[0133] Optionally, the first camera module and each second camera module synchronously capture the anchor point image.

[0134] Optionally, the exposure time for the first camera module and each second camera module to capture the anchor point image is shorter than the exposure time for capturing the tracking image.

[0135] Optionally, the head-mounted display implements synchronous shooting of anchor images by the first camera module and each second camera module in the following manner: synchronizing the multiple cameras included in the first camera module; determining the shooting time information of the anchor image according to the emission time information of the anchor transmitter; sending the shooting time information of the anchor image to the auxiliary positioning device, the shooting time information of the anchor image used by the head-mounted display is the same as the shooting time information of the anchor image used by the auxiliary positioning device; and controlling the first camera module to shoot the first anchor image according to the shooting time information of the anchor image.

[0136] Optionally, the transmission time information of the anchor point transmitter includes the transmission frequency of the anchor point transmitter, wherein the anchor point transmitter uses a first transmission frequency within a first period of time after the head mounted display is turned on, and uses a second transmission frequency after the first period of time, and the first transmission frequency is greater than the second transmission frequency.

[0137] Optionally, the head-mounted display determines the conversion relationship between the coordinate system of the head-mounted display and the coordinate system of the auxiliary positioning device in the following manner: determining the first relative position information of the anchor point relative to the head-mounted display based on the first anchor point image; determining the second relative position information of the anchor point relative to the auxiliary positioning device based on the second anchor point image or receiving the second relative position information sent by the auxiliary positioning device; determining the conversion relationship based on the second relative position information and the first relative position information of the anchor point.

[0138] After determining the conversion relationship, the head mounted display saves the conversion relationship locally and can use the conversion relationship to perform posture conversion within a certain period of time.

[0139] Optionally, the head-mounted display also sends a first synchronization signal to the auxiliary positioning device, where the first synchronization signal is used to synchronize the time between the auxiliary positioning device and the head-mounted display.

[0140] Optionally, the head-mounted display further includes a wireless chip and a processor, and the head-mounted display achieves time synchronization between the wireless chip and the processor by triggering an interrupt operation and sending a second synchronization signal through the wireless chip.

[0141] Optionally, the head-mounted display sends a first point cloud database to the auxiliary positioning device, the first point cloud database including feature point data of the environment in which the head-mounted display is located; updates the first point cloud database according to the tracking image collected by the first camera module; receives point cloud update data sent by the auxiliary positioning device, and updates the first point cloud database according to the point cloud update data.

[0142] For the specific implementation of this embodiment, please refer to the relevant description in the above embodiments, which will not be repeated here.

[0143] In order to facilitate better implementation of the positioning tracking method of the embodiment of the present application, the embodiment of the present application also provides a positioning tracking device, which can be configured in the head-mounted display of the XR device provided in the above embodiment. Fig.12 A structural diagram of a positioning and tracking device provided in Embodiment 5 of the present application is shown in FIG. Fig.12 As shown, the positioning tracking device 100 may include:

[0144] A positioning module 11, configured to obtain a first positioning result of the head mounted display according to a first tracking image acquired by a first camera module and first IMU data obtained by first IMU measurement;

[0145] A receiving module 12 is used to receive a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, where the second positioning result is determined by the auxiliary positioning device according to a second tracking image collected by a second camera and second IMU data measured by a second IMU;

[0146] An acquisition module 13 is used to acquire a first anchor point image captured by the first camera and a second anchor point image captured by the second camera, wherein the first anchor point image and the second anchor point image include images of light spots emitted by the anchor point emitter;

[0147] A determination module 14, configured to determine a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image;

[0148] The coordinate conversion module 15 is used to determine the position information of the auxiliary positioning device in the coordinate system of the head mounted display according to the conversion relationship and the second positioning result.

[0149] In some implementations, the first camera module and each of the second camera modules are time synchronized when capturing images.

[0150] In some implementations, the apparatus 100 further includes a synchronization module, configured to:

[0151] Synchronizing a plurality of cameras included in the first camera module;

[0152] Determining the shooting time information of the anchor point image according to the emission time information of the anchor point transmitter;

[0153] Sending shooting time information of the anchor image to the auxiliary positioning device, wherein the shooting time information of the anchor image used by the head mounted display is the same as the shooting time information of the anchor image used by the auxiliary positioning device;

[0154] The acquisition module 13 is specifically used to control the first camera module to capture the first anchor point image according to the capturing time information of the anchor point image.

[0155] In some implementations, the exposure time for the first camera module and each of the second camera modules to capture the anchor point image is shorter than the exposure time for capturing the tracking image.

[0156] In some implementations, the transmission time information of the anchor point transmitter includes the transmission frequency of the anchor point transmitter, wherein the anchor point transmitter uses a first transmission frequency within a first period of time after the head-mounted display is turned on, and uses a second transmission frequency after the first period of time, and the first transmission frequency is greater than the second transmission frequency.

[0157] In some implementations, the determining module 14 is specifically configured to:

[0158] Determining first relative position information of the anchor point relative to the head mounted display according to the first anchor point image;

[0159] Determine, according to the second anchor point image, second relative position information of the anchor point relative to the auxiliary positioning device, or receive the second relative position information sent by the auxiliary positioning device, where the second relative position information is determined by the auxiliary positioning device according to the second anchor point image;

[0160] The conversion relationship is determined according to the second relative position information and the first relative position information of the anchor point.

[0161] In some implementations, a sending module is further included, which is used to: send a first synchronization signal to the auxiliary positioning device, where the first synchronization signal is used to synchronize the time of the auxiliary positioning device and the head-mounted display.

[0162] In some implementations, the head-mounted display further includes a wireless chip and a processor, and the device 100 further includes a synchronization module for: achieving time synchronization between the wireless chip and the processor through an interrupt operation triggered by the wireless chip and a second synchronization signal sent.

[0163] In some implementations, the point cloud processing module is used to:

[0164] Sending a first point cloud database to the auxiliary positioning device, wherein the first point cloud database includes feature point data of the environment in which the head mounted display is located;

[0165] updating the first point cloud database according to the tracking image collected by the first camera module;

[0166] Receive the point cloud update data sent by the auxiliary positioning device, and update the first point cloud database according to the point cloud update data.

[0167] In some implementations, the auxiliary positioning device includes a handle and a body tracker worn on different body parts.

[0168] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here.

[0169] The above describes the device 100 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0170] The present application also provides an XR device, including: a head-mounted display and an auxiliary positioning device, wherein the head-mounted display includes a first camera module, a first IMU and an anchor transmitter, and the anchor transmitter is used to emit a light spot; the auxiliary positioning device includes a second camera module and a second IMU. The head-mounted display and the auxiliary positioning device are used to perform the positioning and tracking method described in the above method embodiment. The structure of the head-mounted display and the auxiliary positioning device refers to Figure 2As shown, no further description is given here.

[0171] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0172] The present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the corresponding process in the human motion capture method based on sparse IMU in the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0174] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0175] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A positioning tracking method, characterized in that: Applied to an extended reality XR device, the XR device includes a head-mounted display and an auxiliary positioning device, the head-mounted display includes a first camera module, a first inertial measurement unit IMU and an anchor point transmitter, the auxiliary positioning device includes a second camera module and a second IMU, and the anchor point transmitter is used to emit a light spot; the method includes: The head mounted display obtains a first positioning result of the head mounted display according to a first tracking image acquired by the first camera module and first IMU data measured by the first IMU; The auxiliary positioning device obtains a second positioning result of the auxiliary positioning device according to the second tracking image collected by the second camera module and the second IMU data measured by the second IMU, and sends the second positioning result to the head mounted display; The first camera acquires a first anchor point image, and the second camera acquires a second anchor point image corresponding to the first anchor point image, wherein the first anchor point image and the second anchor point image include images of the light spot emitted by the anchor point emitter; The head mounted display determines, according to the first anchor point image and the second anchor point image, a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device; According to the conversion relationship and the second positioning result, the position and posture information of the auxiliary positioning device in the coordinate system of the head mounted display is determined.

2. The method according to claim 1, characterized in that The first camera module and each of the second camera modules are time synchronized when capturing images.

3. The method according to claim 2, characterized in that The method further comprises: The head mounted display synchronizes a plurality of cameras included in the first camera module; The head mounted display determines the shooting time information of the anchor point image according to the emission time information of the anchor point transmitter; The head mounted display sends shooting time information of the anchor point image to the auxiliary positioning device, and the shooting time information of the anchor point image used by the head mounted display is the same as the shooting time information of the anchor point image used by the auxiliary positioning device; The first camera acquires a first anchor point image, and the second camera acquires a second anchor point image corresponding to the first anchor point image, including: The head mounted display controls the first camera module to capture the first anchor point image according to the capturing time information of the anchor point image; The auxiliary positioning device controls the second camera module to capture the second anchor point image according to the capturing time information of the anchor point image.

4. The method according to claim 2, characterized in that: The exposure time for the first camera module and the second camera module to capture the anchor point image is shorter than the exposure time for capturing the tracking image.

5. The method according to claim 3, characterized in that: The transmission time information of the anchor point transmitter includes a transmission frequency, wherein the anchor point transmitter adopts a first transmission frequency within a first time period after the head mounted display is turned on, and adopts a second transmission frequency after the first time period, and the first transmission frequency is greater than the second transmission frequency.

6. The method according to any one of claims 1 to 5, characterized in that: The head mounted display determines a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image, including: The head mounted display determines first relative position information of the anchor point relative to the head mounted display according to the first anchor point image; The head mounted display determines second relative position information of the anchor point relative to the auxiliary positioning device according to the second anchor point image, or the head mounted display receives the second relative position information sent by the auxiliary positioning device, where the second relative position information is determined by the auxiliary positioning device according to the second anchor point image; The head mounted display determines the conversion relationship according to the second relative position information and the first relative position information of the anchor point.

7. The method according to any one of claims 1 to 5, characterized in that: Also includes: The head mounted display sends a first synchronization signal to the auxiliary positioning device; The auxiliary positioning device achieves time synchronization with the head mounted display according to the first synchronization signal.

8. The method according to claim 7, characterized in that The head mounted display further includes a wireless chip and a processor, and the method further includes: The wireless chip and the processor achieve time synchronization through interrupt operation and a second synchronization signal.

9. The method according to any one of claims 1 to 5, characterized in that: The head mounted display obtains a first positioning result of the head mounted display according to the first tracking image acquired by the first camera module and the first IMU data measured by the first IMU, including: The head mounted display obtains the first positioning result according to the first tracking image, the first IMU data, and a first point cloud database of the head mounted display, where the first point cloud database includes feature point data of an environment where the head mounted display is located; The auxiliary positioning device obtains a second positioning result of the auxiliary positioning device according to the second tracking image collected by the second camera module and the second IMU data measured by the second IMU, including: The auxiliary positioning device obtains the second positioning result according to the second tracking image, the second IMU data and a second point cloud database of the auxiliary positioning device, wherein the second point cloud database includes feature point data of the environment in which the auxiliary positioning device is located.

10. The method according to claim 9, characterized in that The second point cloud database is sent by the head mounted display to the auxiliary positioning device, and the method further includes: The head mounted display updates the first point cloud database according to the tracking image collected by the first camera module; The auxiliary positioning device updates the second point cloud database according to the tracking image collected by the second camera module; The auxiliary positioning device sends the updated data of the second point cloud database to the head mounted display; The head mounted display updates the first point cloud database according to the update data.

11. The method according to any one of claims 1 to 5, characterized in that: The auxiliary positioning device includes a handle and a body tracker worn on different body parts.

12. A positioning tracking method, characterized in that: A head mounted display applied to an XR device, wherein the XR device further comprises an auxiliary positioning device, wherein the head mounted display comprises a first camera module, a first inertial measurement unit (IMU) and an anchor point transmitter, wherein the auxiliary positioning device comprises a second camera module and a second IMU, and the anchor point transmitter is used to emit a light spot; wherein the method comprises: Acquire a first positioning result of the head mounted display according to a first tracking image acquired by the first camera module and first IMU data measured by the first IMU; receiving a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, where the second positioning result is determined by the auxiliary positioning device according to a second tracking image collected by the second camera and second IMU data measured by the second IMU; Acquire a first anchor point image captured by the first camera and a second anchor point image captured by the second camera, wherein the first anchor point image and the second anchor point image include images of light spots emitted by the anchor point emitter; Determining a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image; According to the conversion relationship and the second positioning result, the position and posture information of the auxiliary positioning device in the coordinate system of the head mounted display is determined.

13. The method according to claim 12, characterized in that The first camera module and the second camera module are time synchronized when capturing images.

14. The method according to claim 13, characterized in that The method further comprises: Synchronizing a plurality of cameras included in the first camera module; Determining the shooting time information of the anchor point image according to the emission time information of the anchor point transmitter; Sending shooting time information of the anchor image to the auxiliary positioning device, wherein the shooting time information of the anchor image used by the head mounted display is the same as the shooting time information of the anchor image used by the auxiliary positioning device; According to the shooting time information of the anchor point image, the first camera module is controlled to shoot the first anchor point image.

15. The method according to claim 13, characterized in that The exposure time for the first camera module and each of the second camera modules to capture the anchor point image is shorter than the exposure time for capturing the tracking image.

16. The method according to claim 14, characterized in that The transmission time information of the anchor point transmitter includes a transmission frequency, wherein the anchor point transmitter adopts a first transmission frequency within a first time period after the head mounted display is turned on, and adopts a second transmission frequency after the first time period, and the first transmission frequency is greater than the second transmission frequency.

17. The method according to any one of claims 12 to 16, characterized in that: The step of determining a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image includes: Determining first relative position information of the anchor point relative to the head mounted display according to the first anchor point image; Determine, according to the second anchor point image, second relative position information of the anchor point relative to the auxiliary positioning device, or receive the second relative position information sent by the auxiliary positioning device, where the second relative position information is determined by the auxiliary positioning device according to the second anchor point image; The conversion relationship is determined according to the second relative position information and the first relative position information of the anchor point.

18. The method according to any one of claims 12 to 16, characterized in that: Also includes: A first synchronization signal is sent to the auxiliary positioning device, where the first synchronization signal is used to synchronize the time of the auxiliary positioning device and the head mounted display.

19. The method according to claim 18, characterized in that The head mounted display further includes a wireless chip and a processor, and the method further includes: The time synchronization between the wireless chip and the processor is achieved through the interrupt operation triggered by the wireless chip and the second synchronization signal sent.

20. The method according to any one of claims 12 to 16, characterized in that: The method further comprises: Sending a first point cloud database to the auxiliary positioning device, wherein the first point cloud database includes feature point data of the environment in which the head mounted display is located; updating the first point cloud database according to the tracking image collected by the first camera module; Receive the point cloud update data sent by the auxiliary positioning device, and update the first point cloud database according to the point cloud update data.

21. A positioning tracking device, characterized in that: In a head mounted display of an XR device according to any one of claims 12 to 20, the device comprises: A positioning module, configured to obtain a first positioning result of the head mounted display according to a first tracking image acquired by a first camera module and first IMU data obtained by first IMU measurement; A receiving module, configured to receive a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, wherein the second positioning result is determined by the auxiliary positioning device according to a second tracking image collected by a second camera and second IMU data measured by a second IMU; an acquisition module, configured to acquire a first anchor point image acquired by the first camera and a second anchor point image acquired by the second camera, wherein the first anchor point image and the second anchor point image include images of light spots emitted by the anchor point emitter; a determination module, configured to determine a conversion relationship between a coordinate system of the head mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image; A coordinate conversion module is used to determine the position information of the auxiliary positioning device in the coordinate system of the head mounted display according to the conversion relationship and the second positioning result.

22. An extended reality device, characterized in that: include: A head mounted display, comprising a first camera module, a first inertial measurement unit IMU and an anchor point transmitter, wherein the anchor point transmitter is used to emit a light spot; An auxiliary positioning device, comprising a second camera module and a second IMU; The head mounted display and the auxiliary positioning device are used to execute the method according to any one of claims 1 to 11.

23. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program causing a computer to execute the method according to any one of claims 12 to 20.

24. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 12 to 20 is implemented.

Images