Positioning tracking method, apparatus, device, medium, and program

Abstract

Embodiments of the present application provide a positioning and tracking method, device, equipment, medium and program, a camera module is arranged on each auxiliary positioning device, the auxiliary positioning device realizes self-tracking based on the camera module and the IMU sensor thereof, and the positioning result of the auxiliary positioning device is sent to the head-mounted display. In addition, an anchor point transmitter is also arranged on the head-mounted display, the camera of the head-mounted display and the auxiliary positioning device captures the light spot emitted by the anchor point transmitter to obtain an anchor point image, the head-mounted display aligns the coordinate system of the auxiliary positioning device and the coordinate system of the head-mounted display based on the anchor point image, and the auxiliary positioning device is positioned according to the aligned coordinate system. Since each auxiliary positioning device is positioned based on the image captured by the camera thereof and the IMU data, the auxiliary positioning device is not limited by the field of view of the head-mounted display, and the coordinate system is aligned through the anchor point image, which reduces the tracking error and improves the tracking precision.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the virtual reality technical field, and particularly relate to a positioning tracking method, device, set, medium and program. BACKGROUND

[0002] At present, the application scenarios of extended reality (XR) technology are more and more extensive, and the XR technology specifically includes virtual reality (VR) technology, augmented reality (AR) technology and mixed reality (MR) technology, etc. Through the XR technology, a user can immerse in various virtual scenes.

[0003] In an XR game scenario, a human body motion capture technology is used to detect posture information and motion trajectories of a human body in a three-dimensional space, and human body motion is simulated in a virtual scene. The human body motion capture is also called human body positioning tracking. In the prior art, a camera in a head-mounted display of an integrated XR device is usually used for positioning. The camera continuously collects various environment images in a motion process to determine position information of the head-mounted display. At the same time, an optical tracking method is used to determine a relative position relationship between the head-mounted display and other interactive devices (handheld controllers or trackers), so as to determine position information of each other interactive device.

[0004] However, the optical tracking is limited by a field of view (FOV) of each camera in the head-mounted display, and can only track the relative position relationship between other interactive devices and the head-mounted display within the FOV. When a certain interactive device is outside the FOV, the interactive device cannot be positioned and tracked, so that the positioning and tracking of the integrated VR smart device has certain limitations. SUMMARY

[0005] Embodiments of the present application provide a positioning tracking method, device, equipment, medium and program. Positioning of an auxiliary positioning device of an XR device based on images and IMU data captured by a camera of the auxiliary positioning device is limited by a field of view of a head-mounted display, and a coordinate system is aligned through an anchor point image, tracking error is reduced, and tracking precision is improved.

[0006] In a first aspect, embodiments of the present application provide a positioning tracking method applied to an extended reality (XR) device. The XR device includes a head-mounted display and an auxiliary positioning device. The head-mounted display includes a first camera module, a first inertial measurement unit (IMU) and an anchor point emitter. The auxiliary positioning device includes a second camera module and a second IMU. The anchor point emitter is used to emit a light spot. The method includes:

[0007] The head-mounted display obtains a first positioning result of the head-mounted display according to a first tracking image collected by the first camera module and first IMU data measured by the first IMU;

[0008] The auxiliary positioning device obtains a second positioning result of the auxiliary positioning device according to a second tracking image collected by the second camera module and second IMU data measured by the second IMU, and sends the second positioning result to the head-mounted display;

[0009] The first camera obtains a first anchor point image, and the second camera obtains a second anchor point image corresponding to the first anchor point image, wherein the first anchor point image and the second anchor point image include images of light spots emitted by the anchor point emitter;

[0010] The head-mounted display determines a conversion relationship between a coordinate system of the head-mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image;

[0011] According to the conversion relationship and the second positioning result, the pose information of the auxiliary positioning device in the coordinate system of the head-mounted display is determined.

[0012] In a second aspect, an embodiment of the present application provides a positioning and tracking method applied to a head-mounted display of an XR device, the XR device further comprising an auxiliary positioning device, the head-mounted display comprising a first camera module, a first inertial measurement unit (IMU) and an anchor point emitter, the auxiliary positioning device comprising a second camera module and a second IMU, and the anchor point emitter being configured to emit light spots; the method comprising:

[0013] According to a first tracking image collected by the first camera module and first IMU data measured by the first IMU, a first positioning result of the head-mounted display is obtained;

[0014] The second positioning result of the auxiliary positioning device is received, the second positioning result being determined by the auxiliary positioning device according to a second tracking image collected by the second camera and second IMU data measured by the second IMU;

[0015] A first anchor point image collected by the first camera and a second anchor point image collected by the second camera are obtained, wherein the first anchor point image and the second anchor point image include images of light spots emitted by the anchor point emitter;

[0016] According to the first anchor point image and the second anchor point image, a conversion relationship between a coordinate system of the head-mounted display and a coordinate system of the auxiliary positioning device is determined;

[0017] According to the conversion relationship and the second positioning result, the pose information of the auxiliary positioning device in the coordinate system of the head-mounted display is determined.

[0018] In a third aspect, an embodiment of the present application provides a positioning and tracking device, which is configured in a head-mounted display of an XR device in the second aspect, and the device comprises:

[0019] A positioning module is configured to obtain a first positioning result of the head-mounted display according to a first tracking image collected by a first camera module and first IMU data measured by a first inertial measurement unit (IMU);

[0020] A receiving module is configured to receive a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, the second positioning result being determined by the auxiliary positioning device according to second tracking image collected by a second camera and second IMU data measured by a second IMU;

[0021] An obtaining module is configured to obtain a first anchor point image collected by the first camera and a second anchor point image collected by the second camera, wherein the first anchor point image and the second anchor point image comprise images of light spots emitted by the anchor point emitter;

[0022] A determining module is configured to determine a conversion relationship between a coordinate system of the head-mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image;

[0023] A coordinate conversion module is configured to determine pose information of the auxiliary positioning device in the coordinate system of the head-mounted display according to the conversion relationship and the second positioning result.

[0024] In a fourth aspect, an embodiment of the present application provides an XR device, which comprises:

[0025] A head-mounted display comprises a first camera module, a first inertial measurement unit (IMU) and an anchor point emitter, and the anchor point emitter is configured to emit light spots;

[0026] An auxiliary positioning device comprises a handle and a body tracker worn on different body parts, and the auxiliary positioning device comprises a second camera module and a second IMU;

[0027] The head-mounted display and the auxiliary positioning device are configured to perform the method in the first aspect of the present application.

[0028] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, which is configured to store a computer program, and the computer program is configured to make a computer execute the method in the second aspect.

[0029] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, characterized in that the computer program is executed by a processor to implement the method of the second aspect.

[0030] The positioning and tracking method, device, equipment, medium and program provided by the embodiments of the present application can achieve self-tracking of each auxiliary positioning device based on the camera module and the IMU sensor of the auxiliary positioning device, and send the positioning result of the auxiliary positioning device to the head-mounted display. In addition, the head-mounted display is also configured with an anchor point transmitter. The camera of the head-mounted display and the auxiliary positioning device captures the light spot emitted by the anchor point transmitter to obtain an anchor point image. The head-mounted display aligns the coordinate system of the auxiliary positioning device and the coordinate system of the head-mounted display based on the anchor point image, and determines the pose information of the auxiliary positioning device in the coordinate system of the head-mounted display according to the aligned coordinate system. Since each auxiliary positioning device performs positioning based on the image captured by the camera and the IMU data of the auxiliary positioning device, the auxiliary positioning device does not need to be limited by the field of view of the head-mounted display. In addition, the coordinate system is aligned through the anchor point image, which reduces the tracking error and improves the tracking accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0032] Figure 1 It is a structural schematic diagram of an XR device in the prior art;

[0033] Figure 2 It is a structural block diagram of an XR device shown by an embodiment of the present application;

[0034] Figure 3 It is a schematic diagram of a light spot emitted by an anchor point transmitter;

[0035] Figure 4 It is a flowchart of a positioning and tracking method provided by an embodiment one of the present application;

[0036] Figure 5 It is a schematic diagram of system time of a head-mounted display and an auxiliary positioning device;

[0037] Figure 6 It is a flowchart of a synchronization method in a positioning and tracking process provided by an embodiment two of the present application;

[0038] Figure 7 It is a synchronization diagram of a wireless chip and a processor of a head-mounted display;

[0039] Figure 8 Schematic diagram of synchronization of the head-mounted display and the auxiliary positioning device;

[0040] Figure 9 Flowchart of the positioning and tracking method provided in Embodiment Three of the present application;

[0041] Figure 10 Schematic diagram of software function modules when the head-mounted display and the auxiliary positioning device are positioned by using the SLAM method;

[0042] Figure 11 Flowchart of the positioning and tracking method provided in Embodiment Four of the present application;

[0043] Figure 12 Structural schematic diagram of a positioning and tracking device provided in Embodiment Five of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0045] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0046] The embodiments of the present application provide a positioning and tracking method, which can be applied to an XR device. The XR device includes a head-mounted display and an auxiliary positioning device, also referred to as a positioning device or an auxiliary positioner. The auxiliary positioning device includes a handle and a body tracker worn on different body parts. The number of handles is usually two, worn on the left and right hands of a user. The number of body trackers can be two or more, worn on the wrist, calf position, or thigh position, waist, etc. The body trackers worn on different body parts are used to track the motion data of different body parts.

[0047] The existing XR device usually adopts optical tracking to position and track the head-mounted display and the auxiliary positioning device, Figure 1 Fig. 1 is a structural schematic diagram of an XR device in the prior art, and various functional modules configured in the head-mounted display and the auxiliary positioning device of the XR device are shown as Figure 1

[0048] The positioning principle of the existing XR device includes that the head-mounted display acquires optical images of the environment where the head-mounted display is located through the configured camera, and light-emitting diode modules are configured on the left and right handles so that the camera module in the head-mounted display can acquire images. Moreover, inertial measurement unit (IMU) sensors are configured on the head-mounted display, the left and right handles, and each body tracker to assist in positioning and tracking on the basis of optical tracking.

[0049] Specifically, the head-mounted display can acquire optical images of the environment where the head-mounted display is located in real time through the camera module, and the positioning and tracking of the head-mounted display are realized according to the optical images and the relevant position information detected by the IMU sensor on the head-mounted display. Then, the relative position relationship between the auxiliary positioning device and the head-mounted display is analyzed according to the position of the auxiliary positioning device in the optical images, and the positioning and tracking of the auxiliary positioning device such as the left and right handles and each body tracker are realized in combination with the relevant position information detected by the IMU sensors on the left and right handles and each body tracker.

[0050] As described above, when the head-mounted display and the auxiliary positioning device of the XR device are positioned and tracked by the optical tracking method, the field of view of the head-mounted display is limited. When a certain auxiliary positioning device is outside the field of view, the acquired optical images cannot represent the motion position of the auxiliary positioning device, and the positioning and tracking of the auxiliary positioning device cannot be realized, so that the positioning and tracking of the XR device have certain limitations.

[0051] ​To solve the above problems, the application adds a camera module and a processor on each auxiliary positioning device on the basis of the existing structure of the XR device. Each auxiliary positioning device can realize self-tracking based on its own camera module and IMU sensor, and send its own positioning information to the head-mounted display. In addition, the head-mounted display is also configured with an anchor point transmitter. The anchor point transmitter is used to emit a light spot. The camera of the head-mounted display and the auxiliary positioning device take pictures of the light spot emitted by the anchor point transmitter at the same time to obtain an anchor point image. The head-mounted display aligns the coordinate system of the auxiliary positioning device with the coordinate system of the head-mounted display based on the anchor point image, and positions the auxiliary positioning device according to the aligned coordinate system. In this positioning method, the auxiliary positioning device is positioned based on the image captured by its own camera and the IMU data, without being limited by the field of view of the head-mounted display. The coordinate system is aligned through the anchor point image, which reduces the tracking error and improves the tracking accuracy.

[0052] Figure 2 A structural block diagram of an XR device is shown in an embodiment of the application. As shown in the figure, the XR includes a head-mounted display, left and right handles, and two body trackers. The left and right handles have the same structure, and the two body trackers have the same structure. The embodiment of the application only describes the functions of the modules related to the positioning and tracking method provided by the application, and the functions of other modules are not described in detail. Figure 2

[0053] The head-mounted display includes a camera module, an IMU sensor, a processor, and an anchor point transmitter. The processor is used to realize 6-degree-of-freedom (Dof) positioning of the head-mounted display according to the tracking image collected by the camera module and the IMU data collected by the IMU sensor. The 6Dof positioning data includes position information and attitude information (referred to as pose information) of the head-mounted display.

[0054] The anchor point transmitter is used to emit a light spot. The number of light spots can be one or more. The camera module of the head-mounted display and the camera module of the auxiliary positioning device take pictures of the light spot emitted by the anchor point transmitter to obtain an anchor point image. The anchor point image includes a light spot. The light spot in the anchor point image is called an anchor point. The head-mounted display aligns the coordinate system of the auxiliary positioning device with the coordinate system of the head-mounted display based on the anchor point image captured by its own camera module and the anchor point image captured by the auxiliary positioning device.

[0055] ​The anchor transmitter can be a structured light sensor or a Time of Flight (ToF) sensor. The structured light is a system structure composed of a projector and a camera. The projector projects specific light information onto the surface of an object and the background, and the camera collects the light signal changes caused by the object to calculate the position and depth of the object, and then restore the entire three-dimensional space. In this embodiment, the anchor transmitter is a structured light projector. The camera module of the head-mounted display and the camera module of the auxiliary positioning device form a structured light with the anchor transmitter. The Tof sensor can emit infrared light to the surface of an object, and the camera can capture the infrared light.

[0056] The handle includes at least a camera module, an IMU sensor, and a processor. The processor is configured to realize 6Dof tracking of the handle based on tracking images collected by the camera module and IMU data collected by the IMU sensor.

[0057] The body tracker includes at least a camera module, an IMU sensor, and a processor. The processor is configured to realize 6Dof tracking of the body tracker based on tracking images collected by the camera module and IMU data collected by the IMU sensor.

[0058] Figure 2 The head-mounted display, the handle, and the body tracker further include a wireless communication module. The handle and the body tracker communicate with the head-mounted display wirelessly through the wireless communication module. The wireless communication can use a 2.4G network transmission protocol or a low-power Bluetooth protocol. It should be understood that the handle and the body tracker can also communicate with the head-mounted display through a wired manner, which is not limited in the present application.

[0059] In the present application, the images captured by the camera modules of the head-mounted display, the handle, and the body tracker include two types of images: tracking images and anchor images. The tracking images are used to track the head-mounted display, the handle, and the body tracker. The tracking images capture images of the real environment in which the head-mounted display, the handle, and the body tracker are located.

[0060] The anchor image refers to an image captured based on a light spot emitted by the anchor transmitter. The light spot emitted by the anchor transmitter can use visible light or invisible light. For example, the anchor transmitter uses a laser to emit multiple light spots. The user cannot see the light spots, but the camera can capture the light spots.

[0061] Figure 3 The schematic diagram of the light spot emitted by the anchor transmitter is as follows: Figure 3As shown, the anchor point transmitter transmits a plurality of light spots, and the camera modules of the head-mounted display, the handle, and the body tracker capture the light spots, respectively. The camera modules of the head-mounted display, the handle, and the body tracker are located at different positions and angles to capture the light spots. The head-mounted display can calculate the relative position relationship between the anchor point and the camera based on the anchor point images captured at different angles and positions. According to the position relationship between the same anchor point and different cameras, the conversion relationship between the coordinate systems of the head-mounted display and the body tracker, and the conversion relationship between the coordinate systems of the head-mounted display and the handle can be obtained. The relative position relationship between the anchor point and the camera can also be understood as the relative position relationship between the anchor point and the device where the camera is located.

[0062] Figure 4 A flowchart of the positioning and tracking method provided for Embodiment One of the present application is shown. The method of the present embodiment is executed by the XR device shown in the figure. In the present embodiment, for the sake of clarity, the camera module and the IMU on the head-mounted display are referred to as the first camera module and the first IMU, the camera module and the IMU on the auxiliary positioning device are referred to as the second camera module and the second IMU, and the tracking image and the anchor point image captured by the head-mounted display are referred to as the first tracking image and the first anchor point image. The tracking image and the anchor point image captured by the auxiliary positioning device are referred to as the second tracking image and the second anchor point image. As shown in the figure, the positioning and tracking method comprises the following steps. Figure 2 Figure 4

[0063] S101, the head-mounted display obtains the first positioning result of the head-mounted display according to the first tracking image collected by the first camera module and the first IMU data measured by the first IMU.

[0064] The first positioning result comprises 6Dof positioning data of the head-mounted display. The 6Dof positioning data comprises position movement information (3 degrees of freedom) on the X, Y, and Z axes and rotation information on the X, Y, and Z axes. Therefore, not only the change in the field of view angle caused by the rotation of the body part can be detected, but also the change in the field of view caused by the movement of the body part can be detected.

[0065] In the present embodiment, the head-mounted display can use a Simultaneous Localization and Mapping (SLAM) positioning method for positioning. The SLAM positioning method analyzes the position of each environmental feature point in the tracking image relative to the head-mounted display to determine the current position information of the head-mounted display. The rotation information of the head-mounted display is obtained based on the first IMU data measured by the first IMU, thereby obtaining 6Dof tracking data.

[0066] ​​S102, the auxiliary positioning device obtains a second positioning result of the auxiliary positioning device according to the second tracking image collected by the second camera module and the second IMU data measured by the second IMU, and sends the second positioning result to the head-mounted display.

[0067] By arranging a camera on the auxiliary positioning device, the auxiliary positioning device can also use the SLAM positioning method for positioning. The SLAM positioning is a self-tracking method, that is, the device can realize positioning by relying on its own components without the assistance of other devices. The second positioning result obtained by the auxiliary positioning device is 6Dof positioning data of the auxiliary positioning device.

[0068] It should be noted that the handle can be positioned by not only the SLAM positioning method, but also other methods, such as infrared positioning. The handle emits a light spot through a light-emitting unit, and the head-mounted display camera captures the light spot emitted by the light-emitting unit to perform positioning.

[0069] S103, the first camera obtains a first anchor point image, and the second camera obtains a second anchor point image corresponding to the first anchor point image, wherein the first anchor point image and the second anchor point image include images of light spots emitted by an anchor point emitter.

[0070] The anchor point emitter can periodically emit light spots, and the head-mounted display and the auxiliary positioning device collect anchor point images at fixed times. The first anchor point image and the second anchor point image are anchor point images captured at the same time, and the head-mounted display determines the conversion relationship between the coordinate systems of the head-mounted display and the auxiliary positioning device based on the anchor point images captured at the same time. Generally, the collection frequency of the anchor point images is lower than the collection frequency of the tracking images. Generally, 30 frames or 60 frames of tracking images are collected per second, while the anchor point images can be collected once per second, or once per minute, or once for a longer time.

[0071] The first anchor point image and the second anchor point image used to determine the conversion relationship are images of light spots captured at the same time. In order to improve the accuracy of the coordinate system conversion, the first camera module and each second camera module synchronously capture the anchor point images, that is, the first camera module and the second camera module are time-synchronized when capturing images, so as to ensure that the first anchor point image and the second anchor point image are images captured at the same time.

[0072] Optionally, the exposure time of the first camera module and the second camera module for collecting the anchor point images is less than the exposure time of collecting the tracking images. By reducing the exposure time of the camera when capturing the anchor point images, the influence of the environment on the anchor point images can be reduced.

[0073] Since the intrinsic parameters of each camera module are different, the exposure time used by the first camera module and the second camera module when capturing the anchor point images is different.

[0074] S104, the head-mounted display determines a conversion relationship between the coordinate system of the head-mounted display and the coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image.

[0075] In order to reduce the power consumption of the device, the head-mounted display can periodically determine the conversion relationship between the coordinate system of the head-mounted display and the coordinate system of the auxiliary positioning device according to the anchor point image, so as to realize the alignment of the coordinate systems of the head-mounted display and the auxiliary positioning device. The conversion relationship between the coordinate system of the head-mounted display and the coordinate system of the auxiliary positioning device can be a conversion matrix.

[0076] The head-mounted display and the auxiliary positioning device both have their own coordinate systems, and therefore, it is necessary to unify the positioning data of the auxiliary positioning device to the coordinate system of the head-mounted display, that is, to convert the positioning data of the auxiliary positioning device from the coordinate system of the auxiliary positioning device to the coordinate system of the head-mounted display. The conversion of the coordinate systems is also called the alignment of the coordinate systems.

[0077] S105, determining the pose information of the auxiliary positioning device in the coordinate system of the head-mounted display according to the conversion relationship and the second positioning result.

[0078] The conversion relationship of the coordinate systems of the head-mounted display and different auxiliary positioning devices is different. After the head-mounted display determines the conversion relationship of the coordinate systems of the head-mounted display and different auxiliary positioning devices, the head-mounted display saves the conversion relationship of the coordinate systems of the head-mounted display and different auxiliary positioning devices. When the head-mounted display obtains the positioning result based on each frame of tracking image, the positioning result of the auxiliary positioning device is converted based on the conversion relationship, so as to obtain the positioning result of the auxiliary positioning device in the coordinate system of the head-mounted display, that is, the pose information of the auxiliary positioning device in the coordinate system of the head-mounted display.

[0079] After the second positioning result of the auxiliary positioning device is converted to the coordinate system of the head-mounted display, the positioning results of the head-mounted display, the handle and the body tracker in the coordinate system of the head-mounted display are obtained. According to the positioning results of the body tracker, the handle and the head-mounted display, the corresponding user motion posture can be recognized. The motion posture can include user actions and gestures. The tracking manner is no longer limited by the field of view angle of the head-mounted display in the XR device, and the body tracker and the handle can be accurately tracked.

[0080] In the embodiment, a camera module and a processor are added to the auxiliary positioning device of the XR device. The auxiliary positioning device can realize self-tracking based on the camera module and the IMU sensor thereof, and send the positioning result of the auxiliary positioning device to the head-mounted display. In addition, the head-mounted display is also provided with an anchor point transmitter. The camera of the head-mounted display and the auxiliary positioning device captures the light spot emitted by the anchor point transmitter to obtain an anchor point image. The head-mounted display aligns the coordinate system of the auxiliary positioning device and the coordinate system of the head-mounted display based on the anchor point image, and positions the auxiliary positioning device according to the aligned coordinate system. Since each auxiliary positioning device is positioned based on the image captured by the camera thereof and the IMU data, the auxiliary positioning device is not limited by the field of view of the head-mounted display. In addition, the coordinate system is aligned through the anchor point image, the tracking error is reduced, and the tracking accuracy is improved.

[0081] On the basis of the first embodiment, in order to improve the tracking accuracy, the entire device needs to be kept synchronous. On the basis of the device synchronization, the head-mounted display and the multiple camera modules of the auxiliary positioning device can simultaneously capture the anchor point image. Figure 5 The system time of the head-mounted display and the auxiliary positioning device is shown in FIG. 1. Figure 5 As shown in FIG. 1, the processor, the wireless chip, the left handle, the right handle, the body tracker 1 and the body tracker 2 of the head-mounted display each use an independent system time. The system times of the devices are not synchronous. The final goal of the synchronization is that the system time T of the processor of the head-mounted display, the system time T1 of the wireless chip of the head-mounted display, the system time T2 of the left handle, the system time T3 of the right handle, the system time T4 of the body tracker 1 and the system time T5 of the body tracker 2 are completely synchronous, and the synchronization error is required to be within 5us.

[0082] Figure 6 The flowchart of the synchronization method in the positioning and tracking process provided by the second embodiment of the application is shown in FIG. 2. Figure 5 Figure 6 The method provided by the embodiment includes the following steps.

[0083] S201, the wireless chip and the processor of the head-mounted display realize time synchronization through an interruption operation and a second synchronization signal.

[0084] The head-mounted display includes a wireless chip and a processor. The time T1 of the wireless chip is synchronized to the time T of the processor. Figure 7 The synchronization diagram of the wireless chip and the processor of the head-mounted display is shown in FIG. 3. Figure 7 ​As shown, the synchronization of the wireless chip and the processor is completed by means of an interrupt and data communication. Specifically, the wireless chip reads the local time t1, and at the same time, triggers an interrupt by means of hardware. The processor triggers an internal timing t at the time of the interrupt. The wireless chip sends t1 to the processor through a communication interface. The signal carrying t1 is the second synchronization signal. When the processor receives t1, it compares t1 and t to obtain the difference Δt1 between the two. Through Δt1, the time of the wireless chip can be converted into the time of the processor.

[0085] S202, the head-mounted display sends a first synchronization signal to the auxiliary positioning device, and the auxiliary positioning device realizes time synchronization with the head-mounted display according to the first synchronization signal.

[0086] Step S201 realizes the synchronization of T1 and T. This step realizes the synchronization of T1 and T2, T3, T4, and T5. Figure 8 A schematic diagram for the synchronization of the head-mounted display and the auxiliary positioning device is shown in FIG. 2. Figure 8 As shown, the head-mounted display has a global timer: global timer 1. When the application is performing the synchronization function, after the data is ready for sending, the global timer 1 automatically fills the local time t1 into the first synchronization signal in the data sending buffer. The wireless modulation module of the head-mounted display sends the first synchronization signal carrying t1 to the auxiliary positioning device by means of wireless transmission.

[0087] After the auxiliary positioning device is wirelessly demodulated, the global timer 2 of the auxiliary positioning device automatically fills the local time t2 into the data buffer of the application. t2 is the receiving time of the first synchronization signal. The application of the auxiliary positioning device obtains the difference Δt2 between t1 and t2. Through Δt2, the time of the auxiliary positioning device and the time of the head-mounted display can be synchronized.

[0088] It should be noted that the first synchronization signal of the head-mounted display and the auxiliary positioning device has no retransmission mechanism. That is, if the transmission of the first synchronization signal fails in the current synchronization period, the first synchronization signal will not be retransmitted, but will be waited for the next synchronization period, so as to ensure that the time interval Δt2 of wireless modulation and wireless demodulation is a fixed value.

[0089] S203, the head-mounted display synchronizes the plurality of cameras included in the first camera module.

[0090] Through steps S201 and S202, the time synchronization of the processor and the wireless chip of the head-mounted display is realized, and the system time synchronization of the head-mounted display and other auxiliary positioning devices is realized. On the basis of the system time synchronization of the device, the cameras of the head-mounted display and the auxiliary positioning devices are controlled to synchronously capture the anchor point images.

[0091] When the head-mounted display includes multiple cameras, it is necessary to ensure that the multiple cameras work synchronously, that is, synchronously capture tracking images and anchor point images. Due to the different positions of each camera, the corresponding external environment of each camera will be different. In order to ensure the tracking accuracy and the consistency of each optical image output in different environments, the exposure parameters of each camera are also different. For example, the exposure setting of the camera in a darker environment is longer, and the exposure setting of the camera in a better environment is shorter. In this case, the exposure center points of the optical images collected by each camera can be aligned, thereby ensuring that the multiple cameras capture tracking images and anchor point images at the same time, ensuring tracking accuracy and coordinate system alignment accuracy.

[0092] In S204, the head-mounted display determines the capture time information of the anchor point image according to the emission time information of the anchor point emitter.

[0093] The emission time information of the anchor point emitter can be the emission frequency of the anchor point emitter. The emission frequency represents the number of times of emission of the light spot per unit time. The greater the emission frequency, the greater the number of emissions, and the smaller the emission interval. The emission time information of the anchor point emitter can also be the emission interval and the starting emission time of the anchor point emitter.

[0094] The anchor point emitter emits the light spot at a fixed time according to the emission time information. Since the emission time of the light spot emitted by the anchor point emitter is determined, the capture time information of the anchor point image can be determined according to the emission time of the anchor point emitter, and the camera is controlled to capture the anchor point image at a fixed time.

[0095] The capture time information of the anchor point image can be the capture frequency of the anchor point image. The capture frequency can be understood as the number of captures per unit time. The higher the capture frequency per unit time, the shorter the capture interval, that is, the shorter the capture interval of the camera module for the anchor point image. The capture time information of the anchor point image can also be the capture interval and the capture starting time.

[0096] The head-mounted display determines the capture time information of the anchor point image according to the emission time information of the anchor point emitter, which can ensure that the light spot emitted by the anchor point emitter is captured by the camera.

[0097] Optionally, the capture time information of the anchor point image is the same as the emission time information of the anchor point emitter. Correspondingly, the emission time of the anchor point emitter is the same as the capture time of the anchor point image, or the capture time of the anchor point image covers and is greater than the emission time of the anchor point emitter, so as to ensure that the camera can capture the anchor point image.

[0098] In an optional implementation, the anchor point transmitter can emit the light spot at a fixed emission frequency, which is pre-configured. The emission frequency can be set higher if the device or application environment has a high requirement on tracking accuracy, or set lower if the device or application environment has a low requirement on tracking accuracy. The emission frequency also needs to be considered in terms of energy consumption for coordinate system alignment while meeting the tracking accuracy.

[0099] In another optional implementation, the anchor point transmitter uses a first emission frequency in a first time period after the head-mounted display is powered on, and uses a second emission frequency after the first time period. The first emission frequency is greater than the second emission frequency.

[0100] This is because, when the head-mounted display and the auxiliary positioning device are powered on, the head-mounted display and the auxiliary positioning device respectively perform self-tracking, and each uses the feature point data of the environment obtained by itself to form a point cloud database. At this time, the point cloud databases used by the head-mounted display and the auxiliary positioning device are not synchronized, and the point cloud databases are not perfect and accurate. Therefore, the coordinate system alignment cannot be performed using the point cloud databases, and can only be performed using the anchor point images. Therefore, a larger emission frequency can be used for coordinate system alignment to reduce tracking errors and ensure tracking accuracy. Subsequently, as the device operates, the respective point cloud databases gradually become perfect, and the head-mounted display can also share its point cloud database with the auxiliary positioning device to achieve consistency of the point cloud databases. At this time, the coordinate alignment can be performed using the point cloud databases. Therefore, a smaller emission frequency can be used for coordinate system alignment.

[0101] S205, the head-mounted display sends the shooting time information of the anchor point image to the auxiliary positioning device.

[0102] The shooting time information of the anchor point image used by the head-mounted display is the same as the shooting time information of the anchor point image used by the auxiliary positioning device, so that the head-mounted display and the auxiliary positioning device can shoot the anchor point image at the same time.

[0103] S206, the head-mounted display controls the first camera module to shoot the first anchor point image according to the shooting time information of the anchor point image, and the auxiliary positioning device controls the second camera module to shoot the second anchor point image according to the shooting time information of the anchor point image.

[0104] The head-mounted display and the auxiliary positioning device shoot based on the same anchor point image shooting time information, so that the shooting time of the first anchor point image and the second anchor point image is the same. The coordinate system alignment based on the time-synchronized first anchor point image and the second anchor point image can improve the tracking accuracy.

[0105] In this embodiment, after the head-mounted display is time-synchronized with the auxiliary positioning device, the head-mounted display determines the shooting time information of the anchor point image according to the transmission time information of the anchor point transmitter, and sends the shooting time information of the anchor point image to the auxiliary positioning device; the head-mounted display controls the first camera module to shoot the first anchor point image according to the shooting time information of the anchor point image; and the auxiliary positioning device controls the second camera module to shoot the second anchor point image according to the shooting time information of the anchor point image. The shooting time information of the anchor point image used by the head-mounted display is the same as the shooting time information of the anchor point image used by the auxiliary positioning device, so that the head-mounted display and the auxiliary positioning device synchronously shoot the anchor point image, the coordinate system is aligned based on the synchronously shot anchor point image, and the tracking accuracy is improved.

[0106] On the basis of the first embodiment and the second embodiment, the third embodiment of the present application provides a positioning and tracking method for describing step S104 in the first embodiment, that is, determining the conversion relationship between the coordinate systems of the head-mounted display and the auxiliary positioning device according to the first anchor point image collected by the first camera and the second anchor point image collected by the second camera. Figure 9 The flowchart of the positioning and tracking method provided in the third embodiment of the present application is shown in FIG. 1, and the method provided in the present embodiment includes the following steps: Figure 9

[0107] S1041, the head-mounted display determines the first relative position information of the anchor point relative to the head-mounted display according to the first anchor point image.

[0108] The anchor point transmitter transmits a plurality of light spots at a fixed time according to the configured transmission time information, the first camera module shoots the first anchor point image according to the shooting time information of the anchor point image, and the second camera module shoots the second anchor point image according to the shooting time information of the anchor point image.

[0109] The first relative position information is the 3D coordinates of the anchor point in the coordinate system of the head-mounted display, the first anchor point image includes a plurality of anchor points, the head-mounted display determines the 3D coordinates of each anchor point image according to the first anchor point image, and the 3D coordinates of the anchor point determined by the head-mounted display based on the first anchor point image are the 3D coordinates of the anchor point in the coordinate system of the head-mounted display, not the absolute coordinates of the anchor point in the world coordinate system. Therefore, the 3D coordinates of the anchor point determined by the head-mounted display are the relative position of the anchor point relative to the head-mounted display.

[0110] S1042, the auxiliary positioning device determines the second relative position information of the anchor point relative to the auxiliary positioning device according to the second anchor point image, and sends the second relative position information to the head-mounted display.

[0111] ​The second relative position information is 3D coordinates of the anchor point in the coordinate system of the auxiliary positioning device, and the auxiliary positioning device obtains the 3D coordinates of the anchor point in the coordinate system of the auxiliary positioning device by using the same method as the head-mounted display.

[0112] In S1043, the head-mounted display determines the conversion relationship between the coordinate system of the head-mounted display and the coordinate system of the auxiliary positioning device according to the second relative position information and the first relative position information of the anchor point sent by the auxiliary positioning device.

[0113] It can be understood that the head-mounted display needs to use the first relative position information and the second relative position information of the same anchor point when determining the conversion relationship between the head-mounted display and the auxiliary positioning device. Due to the different positions of the head-mounted display and the auxiliary positioning device, the number of anchor points captured by the head-mounted display and the auxiliary positioning device can be the same or different. For example, the anchor point emitter emits 10 light spots, the head-mounted display detects 10 anchor points, and the auxiliary positioning device detects 8 anchor points. Then, the conversion relationship can be determined only by using all or part of the 8 anchor points detected by the auxiliary positioning device.

[0114] When there are multiple auxiliary positioning devices, the head-mounted display determines the conversion relationship between the coordinate system of the head-mounted display and the coordinate system of each auxiliary positioning device.

[0115] Optionally, in other embodiments of the present application, the auxiliary positioning device can also not detect the second anchor point information, but send the second anchor point image to the head-mounted display. The head-mounted display determines the second relative position information of the anchor point relative to the auxiliary positioning device according to the second anchor point image, and determines the first relative position information of the anchor point relative to the head-mounted display according to the first anchor point image. The conversion matrix of the coordinate system of the auxiliary positioning device and the coordinate system of the head-mounted display is determined according to the first relative position and the second relative position of the anchor point.

[0116] In the present embodiment, the head-mounted display, the handle and the body tracker can all use the SLAM positioning method for positioning. Figure 10 The software function module diagram for the head-mounted display and the auxiliary positioning device using the SLAM method for positioning is shown in FIG. 1. Figure 10

[0117] The point cloud data module of the head-mounted display is used to store or maintain the feature point data of the environment in which the head-mounted display is located, that is, the feature point data of the environment in which the head-mounted display is currently using for 6Dof tracking. The feature point data of the environment is also called point cloud data, and the feature point data includes 3D coordinates of the feature point, and can also include depth information of the feature point, descriptors of the feature point, etc. The feature point data of the environment in which the head-mounted display is located can be stored in the first point cloud database in the point cloud data module. The point cloud data module of the head-mounted display can determine the position of the environment in which the head-mounted display is located through the feature point data.

[0118] ​The point cloud data updating module of the head-mounted display collects more environmental feature points during the use of the head-mounted display, and updates and perfects the point cloud data of the environment in real time, and updates the local point cloud data module. The handle point cloud wireless sharing module of the head-mounted display mainly includes two basic functions: the first function is to update the local point cloud data module to the auxiliary positioning device during the boot and non-use stage, so that the auxiliary positioning device can track the position based on the same point cloud data and the same world coordinate system of the head-mounted display; the second function is to complete the point cloud data update of the two systems through the module after the point cloud data of the head-mounted display and the auxiliary positioning device is updated.

[0119] The point cloud database of the head-mounted display includes a plurality of different point cloud databases, and different point cloud databases correspond to different environments. For example, the point cloud database includes three point cloud databases, which are the point cloud databases of room 1, room 2 and room 3. After the head-mounted display is powered on, the head tracking module collects feature point data of the current environment, the point cloud database selection module matches the feature point data of the current environment with the point cloud database data, and updates the matched point cloud database to the point cloud data module. After the point cloud data is updated, the point cloud data updating module also updates the point cloud database.

[0120] The head tracking module combines the data provided by the camera module and the IMU sensor module of the head-mounted display and the point cloud data module to complete the position tracking of the head-mounted display in the current environment.

[0121] The functions of the modules of the auxiliary positioning device are similar to those of the head-mounted display, which will not be repeated here. The auxiliary positioning device shown in the figure is less than the point cloud database, the point cloud database selection and the anchor point emitting module relative to the head-mounted display.

[0122] The head-mounted display obtains a first positioning result according to a first tracking image, first IMU data and a first point cloud database of the head-mounted display, and the first point cloud database includes feature point data of an environment in which the head-mounted display is located. The auxiliary positioning device obtains a second positioning result according to a second tracking image, second IMU data and a second point cloud database of the auxiliary positioning device, and the second point cloud database includes feature point data of an environment in which the auxiliary positioning device is located.

[0123] In an implementation manner, after the head-mounted display and the auxiliary positioning device are powered on, each determines its own point cloud database through a tracking image, and constantly updates and perfects its own point cloud database.

[0124] In another implementation, before the head-mounted display is powered on and runs, if there is no matched first point cloud database in the point cloud database of the head-mounted display, the head-mounted display and the auxiliary positioning device each determine their own point cloud database by tracking images, and constantly update and improve their own point cloud database.

[0125] Optionally, before the head-mounted display is powered on and runs, if there is a matched first point cloud database in the point cloud database of the head-mounted display, the head-mounted display can send the first point cloud database to the auxiliary positioning device. Subsequently, the head-mounted display updates the first point cloud database according to the tracking images collected by the first camera module, and the auxiliary positioning device updates the second point cloud database according to the tracking images collected by the second camera module, that is, the head-mounted display synchronizes the point cloud database to the auxiliary positioning device after being powered on, and subsequently, the head-mounted display and the auxiliary positioning device each update their own point cloud database.

[0126] Optionally, the auxiliary positioning device can also send the update data of the second point cloud database to the head-mounted display, and the head-mounted display updates the first point cloud database according to the update data. The first point cloud database maintained on the head-mounted display is the most complete point cloud database, and in order to improve the tracking accuracy, the head-mounted display can also share the first point cloud database to the auxiliary positioning device regularly.

[0127] The above method embodiment describes the positioning and tracking method of the XR device from the perspective of the interaction between the head-mounted display and the auxiliary positioning devices. Embodiment four of the present application describes the positioning and tracking method from the perspective of the head-mounted display, Figure 11 The flowchart of the positioning and tracking method provided for embodiment four of the present application, the method of the present embodiment is executed by the head-mounted display shown in the above, as shown in the above, Figure 11 The method provided by the present embodiment includes the following steps.

[0128] S301, obtaining a first positioning result of the head-mounted display according to a first tracking image collected by a first camera module and first IMU data measured by a first IMU.

[0129] S302, receiving a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, the second positioning result being determined by the auxiliary positioning device according to a second tracking image collected by a second camera and second IMU data measured by a second IMU.

[0130] S303, obtaining a first anchor point image collected by the first camera and a second anchor point image collected by the second camera, wherein the first anchor point image and the second anchor point image include images of light spots emitted by an anchor point emitter.

[0131] S304, determine the conversion relationship between the coordinate system of the head-mounted display and the coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image.

[0132] S305, determine the pose information of the auxiliary positioning device in the coordinate system of the head-mounted display according to the conversion relationship and the second positioning result of the auxiliary positioning device.

[0133] Optionally, the first camera module and each second camera module synchronously capture the anchor point image.

[0134] Optionally, the exposure time of the first camera module and each second camera module collecting the anchor point image is less than the exposure time of collecting the tracking image.

[0135] Optionally, the head-mounted display synchronously captures the anchor point image by the first camera module and each second camera module in the following way: synchronizing a plurality of cameras included in the first camera module; determining the shooting time information of the anchor point image according to the emission time information of the anchor point emitter; sending the shooting time information of the anchor point image to the auxiliary positioning device, the shooting time information of the anchor point image used by the head-mounted display is the same as the shooting time information of the anchor point image used by the auxiliary positioning device; controlling the first camera module to capture the first anchor point image according to the shooting time information of the anchor point image.

[0136] Optionally, the emission time information of the anchor point emitter includes the emission frequency of the anchor point emitter, wherein the anchor point emitter adopts a first emission frequency within a first time length after the head-mounted display is powered on, and adopts a second emission frequency after the first time length, the first emission frequency is greater than the second emission frequency.

[0137] Optionally, the head-mounted display determines the conversion relationship between the coordinate system of the head-mounted display and the coordinate system of the auxiliary positioning device in the following way: determining the first relative position information of the anchor point relative to the head-mounted display according to the first anchor point image; determining the second relative position information of the anchor point relative to the auxiliary positioning device according to the second anchor point image, or receiving the second relative position information sent by the auxiliary positioning device; determining the conversion relationship according to the second relative position information and the first relative position information of the anchor point.

[0138] After the head-mounted display determines the conversion relationship, the head-mounted display saves the conversion relationship locally, and can use the conversion relationship for pose conversion within a certain time.

[0139] Optionally, the head-mounted display also sends a first synchronization signal to the auxiliary positioning device, the first synchronization signal is used to synchronize the time of the auxiliary positioning device and the head-mounted display.

[0140] Optionally, the head-mounted display further comprises a wireless chip and a processor, and the head-mounted display implements time synchronization between the wireless chip and the processor through the wireless chip triggering the interrupt operation and the second synchronization signal sent.

[0141] Optionally, the head-mounted display sends a first point cloud database to the auxiliary positioning device, the first point cloud database comprising feature point data of an environment in which the head-mounted display is located; the first point cloud database is updated according to a tracking image collected by the first camera module; and point cloud update data sent by the auxiliary positioning device is received, and the first point cloud database is updated according to the point cloud update data.

[0142] The specific implementation of the embodiment is described in the foregoing embodiment, which will not be described here.

[0143] To better implement the positioning and tracking method of the embodiment of the application, the embodiment of the application further provides a positioning and tracking device which can be configured in the head-mounted display of the XR device provided in the foregoing embodiments. Figure 12 As shown in the structural schematic diagram of the positioning and tracking device provided in the fifth embodiment of the application, Figure 12 The positioning and tracking device 100 can comprise:

[0144] The positioning module 11 is configured to obtain a first positioning result of the head-mounted display according to a first tracking image collected by the first camera module and first IMU data measured by the first IMU.

[0145] The receiving module 12 is configured to receive a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, the second positioning result being determined by the auxiliary positioning device according to a second tracking image collected by the second camera and second IMU data measured by the second IMU.

[0146] The obtaining module 13 is configured to obtain a first anchor point image collected by the first camera and a second anchor point image collected by the second camera, wherein the first anchor point image and the second anchor point image comprise images of light spots emitted by the anchor point emitter.

[0147] The determining module 14 is configured to determine a conversion relationship between a coordinate system of the head-mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image.

[0148] The coordinate conversion module 15 is configured to determine pose information of the auxiliary positioning device in the coordinate system of the head-mounted display according to the conversion relationship and the second positioning result.

[0149] In some implementations, the first camera module and each second camera module are time-synchronized when capturing images.

[0150] In some implementations, the apparatus 100 further comprises a synchronization module, configured to:

[0151] synchronize a plurality of cameras comprised in the first camera module;

[0152] determine shooting time information of the anchor point image according to the emission time information of the anchor point emitter;

[0153] send the shooting time information of the anchor point image to the auxiliary positioning apparatus, wherein the shooting time information of the anchor point image used by the head-mounted display is the same as the shooting time information of the anchor point image used by the auxiliary positioning apparatus;

[0154] The acquisition module 13 is specifically configured to: control the first camera module to shoot the first anchor point image according to the shooting time information of the anchor point image.

[0155] In some implementations, the exposure time of the anchor point image collected by the first camera module and each second camera module is less than the exposure time of the tracking image.

[0156] In some implementations, the emission time information of the anchor point emitter comprises an emission frequency of the anchor point emitter, wherein the anchor point emitter adopts a first emission frequency within a first time length after the head-mounted display is powered on, and adopts a second emission frequency after the first time length, the first emission frequency being greater than the second emission frequency.

[0157] In some implementations, the determination module 14 is specifically configured to:

[0158] determine first relative position information of an anchor point relative to the head-mounted display according to the first anchor point image;

[0159] determine second relative position information of the anchor point relative to the auxiliary positioning apparatus according to the second anchor point image, or receive the second relative position information sent by the auxiliary positioning apparatus, the second relative position information being determined by the auxiliary positioning apparatus according to the second anchor point image;

[0160] determine the conversion relationship according to the second relative position information and the first relative position information of the anchor point.

[0161] In some implementations, further comprising a sending module, configured to: send a first synchronization signal to the auxiliary positioning apparatus, the first synchronization signal being used to synchronize the time of the auxiliary positioning apparatus and the head-mounted display.

[0162] In some implementations, the head-mounted display further comprises a wireless chip and a processor, and the apparatus 100 further comprises a synchronization module configured to synchronize the wireless chip and the processor by means of an interrupt operation triggered by the wireless chip and a second synchronization signal sent by the wireless chip.

[0163] In some implementations, the point cloud processing module is configured to:

[0164] send a first point cloud database to the auxiliary positioning apparatus, the first point cloud database comprising feature point data of an environment in which the head-mounted display is located;

[0165] update the first point cloud database according to a tracking image captured by the first camera module;

[0166] receive point cloud update data sent by the auxiliary positioning apparatus, and update the first point cloud database according to the point cloud update data.

[0167] In some implementations, the auxiliary positioning apparatus comprises a handle and a body tracker worn on different body parts.

[0168] It should be understood that the apparatus embodiments and the method embodiments can correspond to each other, and similar descriptions can be made with reference to the method embodiments. To avoid repetition, no further description is given here.

[0169] The apparatus 100 of the embodiments of the present application is described above from the perspective of functional modules in combination with the drawings. It should be understood that the functional modules can be implemented in the form of hardware, or in the form of instructions of software, or in the form of a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits of hardware in a processor and / or instructions of software in the form. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing performed by a processor, or executed by a combination of hardware and software modules in the processor. Alternatively, the software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps in the above method embodiments.

[0170] The embodiments of the present application also provide an XR device, comprising: a head-mounted display and an auxiliary positioning apparatus, the head-mounted display comprising a first camera module, a first IMU and an anchor point transmitter, the anchor point transmitter being configured to emit a light spot; the auxiliary positioning apparatus comprising a second camera module and a second IMU. The head-mounted display and the auxiliary positioning apparatus are configured to perform the positioning and tracking method described in the above method embodiments, and the structure of the head-mounted display and the auxiliary positioning apparatus is described in detail in the above method embodiments. Figure 2As shown, details are not repeated here.

[0171] The application further provides a computer storage medium, which stores a computer program. The computer program is executed by a computer to enable the computer to perform the method of the above method embodiments. Alternatively, the application embodiments further provide a computer program product containing instructions. The instructions are executed by a computer to enable the computer to perform the method of the above method embodiments.

[0172] The application further provides a computer program product, which includes a computer program stored in a computer readable storage medium. A processor of an electronic device reads the computer program from the computer readable storage medium. The processor executes the computer program to enable the electronic device to perform the corresponding procedures in the human motion capture method based on sparse IMU in the embodiments of the application. For brevity, details are not repeated here.

[0173] In several embodiments provided in the application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the above-described device embodiments are merely illustrative. For example, the division of the modules is only a logical function division. There can be another division manner for actual implementation. For example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0174] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical modules, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. For example, the functional modules in the embodiments of the application can be integrated into a processing module, or each module can be physically present alone, or two or more modules can be integrated into one module.

[0175] The above is merely a specific implementation of the application, but the protection scope of the application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A method of position tracking, characterized by, The application is applied to an extended reality (XR) device, the XR device comprising a head-mounted display and an auxiliary positioning device, the head-mounted display comprising a first camera module, a first inertial measurement unit (IMU) and an anchor point transmitter, the auxiliary positioning device comprising a second camera module and a second IMU, the anchor point transmitter being configured to emit a light spot; the method comprising: The head-mounted display obtains a first positioning result of the head-mounted display according to a first tracking image collected by the first camera module and first IMU data measured by the first IMU; The auxiliary positioning device obtains a second positioning result of the auxiliary positioning device according to a second tracking image collected by the second camera module and second IMU data measured by the second IMU, and sends the second positioning result to the head-mounted display; The first camera obtains a first anchor point image, and the second camera obtains a second anchor point image corresponding to the first anchor point image, wherein the first anchor point image and the second anchor point image comprise images of the light spot emitted by the anchor point transmitter; The head-mounted display determines a conversion relationship between a coordinate system of the head-mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image; According to the conversion relationship and the second positioning result, the pose information of the auxiliary positioning device in the coordinate system of the head-mounted display is determined.

2. The method of claim 1, wherein, The first camera module and each second camera module are time-synchronized when capturing images.

3. The method of claim 2, wherein, The method further comprises: The head-mounted display synchronizes a plurality of cameras included in the first camera module; The head-mounted display determines shooting time information of anchor point images according to emission time information of the anchor point transmitter; The head-mounted display sends the shooting time information of the anchor point images to the auxiliary positioning device, and the shooting time information of the anchor point images used by the head-mounted display is the same as the shooting time information of the anchor point images used by the auxiliary positioning device; The first camera obtains a first anchor point image, and the second camera obtains a second anchor point image corresponding to the first anchor point image, comprising: The head-mounted display controls the first camera module to capture the first anchor point image according to the shooting time information of the anchor point images; The auxiliary positioning device controls the second camera module to capture the second anchor point image according to the shooting time information of the anchor point images.

4. The method of claim 2, wherein, The exposure time of the first camera module and the second camera module for capturing anchor point images is less than the exposure time for capturing tracking images.

5. The method of claim 3, wherein, The emission time information of the anchor point transmitter comprises an emission frequency, wherein the anchor point transmitter adopts a first emission frequency within a first time length after the head-mounted display is powered on, and adopts a second emission frequency after the first time length, the first emission frequency being greater than the second emission frequency.

6. The method according to any one of claims 1 to 5, characterized in that, The head-mounted display determines a conversion relationship between a coordinate system of the head-mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image, comprising: The head-mounted display determines first relative position information of the anchor point relative to the head-mounted display according to the first anchor point image; The head-mounted display determines second relative position information of the anchor point relative to the auxiliary positioning device according to the second anchor point image, or the head-mounted display receives the second relative position information sent by the auxiliary positioning device, which is determined by the auxiliary positioning device according to the second anchor point image; The head-mounted display determines the conversion relationship according to the second relative position information and the first relative position information of the anchor point.

7. The method according to any one of claims 1 to 5, characterized in that, Further comprising: The head-mounted display sends a first synchronization signal to the auxiliary positioning device; The auxiliary positioning device realizes time synchronization with the head-mounted display according to the first synchronization signal.

8. The method of claim 7, wherein, The head-mounted display further comprises a wireless chip and a processor, and the method further comprises: The wireless chip and the processor realize time synchronization through interrupt operation and a second synchronization signal.

9. The method according to any one of claims 1 to 5, characterized in that, The head-mounted display obtains a first positioning result of the head-mounted display according to a first tracking image collected by the first camera module and first IMU data measured by the first IMU, comprising: The head-mounted display obtains the first positioning result according to the first tracking image, the first IMU data, and a first point cloud database of the head-mounted display, wherein the first point cloud database comprises feature point data of an environment in which the head-mounted display is located; The auxiliary positioning device obtains a second positioning result of the auxiliary positioning device according to a second tracking image collected by the second camera module and second IMU data measured by the second IMU, comprising: The auxiliary positioning device obtains the second positioning result according to the second tracking image, the second IMU data, and a second point cloud database of the auxiliary positioning device, wherein the second point cloud database comprises feature point data of an environment in which the auxiliary positioning device is located.

10. The method of claim 9, wherein, The second point cloud database is sent by the head-mounted display to the auxiliary positioning device, and the method further comprises: The head-mounted display updates the first point cloud database according to the tracking image collected by the first camera module; The auxiliary positioning device updates the second point cloud database according to the tracking image collected by the second camera module; The auxiliary positioning device sends updated data of the second point cloud database to the head-mounted display; The head-mounted display updates the first point cloud database according to the updated data.

11. The method according to any one of claims 1 to 5, characterized in that, The auxiliary positioning device comprises a handle and a body tracker worn on different body parts.

12. A method of position tracking, characterized by, The head-mounted display applied to an XR device, the XR device further comprising an auxiliary positioning device, the head-mounted display comprising a first camera module, a first inertial measurement unit (IMU), and an anchor point transmitter, the auxiliary positioning device comprising a second camera module and a second IMU, the anchor point transmitter being used to emit a light spot; the method comprising: obtaining a first positioning result of the head-mounted display according to first tracking images collected by the first camera module and first IMU data measured by the first IMU; receiving a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, the second positioning result being determined by the auxiliary positioning device according to second tracking images collected by the second camera and second IMU data measured by the second IMU; obtaining a first anchor point image collected by the first camera and a second anchor point image collected by the second camera, wherein the first anchor point image and the second anchor point image include images of light spots emitted by the anchor point emitter; determining a conversion relationship between a coordinate system of the head-mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image; determining pose information of the auxiliary positioning device in the coordinate system of the head-mounted display according to the conversion relationship and the second positioning result.

13. The method of claim 12, wherein, The first camera module and the second camera module are time-synchronized when shooting images.

14. The method of claim 13, wherein, The method further comprises: synchronizing a plurality of cameras included in the first camera module; determining shooting time information of anchor point images according to emission time information of the anchor point emitter; sending the shooting time information of the anchor point images to the auxiliary positioning device, the shooting time information of the anchor point images used by the head-mounted display being the same as the shooting time information of the anchor point images used by the auxiliary positioning device; controlling the first camera module to shoot the first anchor point image according to the shooting time information of the anchor point images.

15. The method of claim 13, wherein, Exposure times of the first camera module and each second camera module for collecting anchor point images are less than exposure times for collecting tracking images.

16. The method of claim 14, wherein, The emission time information of the anchor point emitter includes an emission frequency, wherein the anchor point emitter uses a first emission frequency within a first time length after the head-mounted display is powered on, and uses a second emission frequency after the first time length, the first emission frequency being greater than the second emission frequency.

17. The method according to any one of claims 12-16, characterized by, The determination of the conversion relationship between the coordinate system of the head-mounted display and the coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image comprises: determining first relative position information of an anchor point relative to the head-mounted display according to the first anchor point image; determining second relative position information of the anchor point relative to the auxiliary positioning device according to the second anchor point image, or receiving the second relative position information sent by the auxiliary positioning device, the second relative position information being determined by the auxiliary positioning device according to the second anchor point image; determining the conversion relationship according to the second relative position information and the first relative position information of the anchor point.

18. The method according to any one of claims 12-16, characterized in that, Further comprising: sending a first synchronization signal to the auxiliary positioning device, the first synchronization signal being used to synchronize time of the auxiliary positioning device and the head-mounted display.

19. The method of claim 18, wherein, The head-mounted display further comprises a wireless chip and a processor, and the method further comprises: The time synchronization of the wireless chip and the processor is realized through the interrupt operation triggered by the wireless chip and the second synchronization signal sent.

20. The method according to any one of claims 12-16, characterized in that, The method further comprises: sending a first point cloud database to the auxiliary positioning device, the first point cloud database comprising feature point data of an environment in which the head-mounted display is located; updating the first point cloud database according to a tracking image collected by the first camera module; receiving point cloud update data sent by the auxiliary positioning device and updating the first point cloud database according to the point cloud update data.

21. A position tracking device, characterized by The device is configured in a head-mounted display of an XR device according to any one of claims 12-20, and the device comprises: a positioning module configured to obtain a first positioning result of the head-mounted display according to a first tracking image collected by a first camera module and first IMU data measured by a first inertial measurement unit (IMU); a receiving module configured to receive a second positioning result of the auxiliary positioning device sent by the auxiliary positioning device, the second positioning result being determined by the auxiliary positioning device according to a second tracking image collected by a second camera and second IMU data measured by a second IMU; an obtaining module configured to obtain a first anchor point image collected by the first camera and a second anchor point image collected by the second camera, wherein the first anchor point image and the second anchor point image comprise images of light spots emitted by the anchor point emitter; a determining module configured to determine a conversion relationship between a coordinate system of the head-mounted display and a coordinate system of the auxiliary positioning device according to the first anchor point image and the second anchor point image; a coordinate conversion module configured to determine pose information of the auxiliary positioning device in the coordinate system of the head-mounted display according to the conversion relationship and the second positioning result.

22. An extended reality device, comprising: The device comprises: a head-mounted display comprising a first camera module, a first inertial measurement unit (IMU), and an anchor point emitter configured to emit light spots; an auxiliary positioning device comprising a second camera module and a second IMU; the head-mounted display and the auxiliary positioning device are configured to perform the method according to any one of claims 1-11.

23. A computer-readable storage medium, characterized in that, A computer program for storing a computer program, the computer program enabling a computer to perform the method according to any one of claims 12-20.

24. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method according to any one of claims 12-20.

Images