Six degree of freedom (6DOF) tracking system and method for mobile head-mounted displays (HMDs)
By integrating computer vision edge models and 6DOF algorithms into the HMD, the positioning error problem of mobile HMDs in dynamic environments is solved, enabling stable virtual reality and augmented reality experiences within vehicles.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- AUDI AG
- Filing Date
- 2021-04-09
- Publication Date
- 2026-06-05
AI Technical Summary
Existing mobile head-mounted displays (HMDs) cannot effectively perform six degrees of freedom (6DOF) in-vehicle and out-of-vehicle tracking in dynamic environments, leading to positioning errors, especially in dynamic environments inside vehicles where the user's viewing position may change suddenly.
By employing a computer vision-based edge model and the 6DOF algorithm, the interior space of the vehicle is observed by a camera in the HMD (Head-Mounted Device) to create an edge model. The application then performs computer vision-based HMD translation calculations to compensate for the positioning error of the 6DOF algorithm, thereby achieving internal and external tracking.
In dynamic environments, ensuring accurate positioning of the HMD and avoiding sudden changes in the user's viewing position enables a stable VR and AR experience within the vehicle.
Smart Images
Figure CN115516506B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a system for detecting at least one mobile head-mounted display (HMD), comprising at least one vehicle and said at least one head-mounted display, wherein the vehicle includes at least one adjustment device and at least one air interface / wireless interface, wherein the adjustment device is designed to provide vehicle sensor data, and said at least one head-mounted display includes at least one camera, a control unit, an air interface and at least one application, wherein the control unit of the head-mounted display is designed to perform in-and-out tracking based on a six-degrees-of-freedom (6DOF) algorithm using at least one camera image recorded / captured by the at least one camera, and to determine the translation of said at least one mobile head-mounted display based on the six-degrees-of-freedom algorithm. Background Technology
[0002] Virtual Reality (VR) and Augmented Reality (AR) are technologies that are now widely used in gaming and production applications. In these applications, users wear what are known as Head-Mounted Displays (HMDs). It is crucial for the accurate positioning of the HMD during use of VR and AR headsets to ensure that content is visually presented correctly to the user. Specifically, this involves determining 6DOF translations along the X, Y, and Z axes, as well as rotations around these axes.
[0003] The location or detection of the HMD is referred to here as tracking. Here, a distinction is made between inside-outside tracking (from the HMD outwards) and outside-inside tracking (from the outside to the HMD). Previously, for stationary applications, outside-inside tracking using external tracking sensors, such as infrared sensors, was existing technology.
[0004] However, future HMD development will focus on mobile HMDs, which do not have external trackers and utilize both internal and external tracking. However, current applications are limited to static environments.
[0005] For example, a method for determining the pose of a camera on a head-mounted screen system in a vehicle is known from document EP 2 491 530 B1. Using an edge model, the camera pose is determined optically from environmental data in six degrees of freedom.
[0006] A system for calibrating a display device worn on a user's head is known from US 20 020 105 484 A1, which is used to display virtual objects in a real environment. The display device has a detection unit that operates in six degrees of freedom for generating an environment model based on image data.
[0007] A device for tracking vehicle occupants using a display unit on a helmet is known from document US 20 100 109 976 A1. The helmet has optical sensors for detecting directional markers within the vehicle's interior space in six degrees of freedom and is suitable for virtual reality applications.
[0008] In existing VR and AR-HMD technologies, tracking algorithms are already implemented in the firmware; however, these assume a static environment. Due to the hardware-dependent implementation of the 6DOF algorithm in the firmware of current HMDs, it is not possible to activate translation for in-and-out tracking. Summary of the Invention
[0009] The purpose of this invention is to provide a system that enables the use of a mobile HMD in dynamic environments.
[0010] This objective is achieved by a system having the features of claim 1. Advantageous improvements are the subject matter of the description and the accompanying drawings.
[0011] This invention relates to a system for detecting at least one mobile head-mounted display (HMD), the system comprising at least one vehicle and said at least one HMD, wherein the vehicle includes at least one adjustment device and at least one air interface, wherein the adjustment device is designed to provide vehicle sensor data, and said at least one HMD includes at least one camera, a control unit, an air interface, and at least one application, wherein the control unit of the HMD is designed to perform in-and-out tracking based on a six-degrees-of-freedom (6DOF) algorithm using camera images recorded by at least one camera, and to determine the translation of said at least one HMD based on the 6DOF algorithm. The at least one HMD is typically designed as glasses. Alternatively, the at least one HMD is designed as a lens. The 6DOF algorithm is integrated into the firmware of the HMD.
[0012] According to the invention, the application is additionally designed to provide an edge model of the vehicle's interior space and, based on this edge model, to provide automated computer vision-based 6DOF tracking of the at least one mobile HMD. Here, tracking refers to the localization or detection of the HMD.
[0013] Computer vision means machine observation. Therefore, this system is designed to enable machine observation of the vehicle's interior space via at least one camera in the HMD. Furthermore, the system is designed to provide an edge model of the interior space.
[0014] This system offers the advantage of incorporating edge models into computer vision-based 6DOF tracking to correct for dynamic environments, such as those passing through vehicle windows, perceived by at least one camera. This prevents erroneous localization of the HMD within the vehicle, which could lead to sudden changes in the user's viewing position within VR and AR content. Here, the user's viewing position is also the viewing position of at least one HMD camera providing computer vision-based observation.
[0015] In one improved embodiment, the application is designed based on computer vision-based 6DOF tracking, enabling computer vision-based HMD translation calculations to be based on an edge model. The application is designed to determine the computer vision-based HMD translation relative to the vehicle's interior space. Using a corresponding edge model of the vehicle's interior space, the system is designed to accurately locate at least one HMD within the vehicle's interior space. Here, the vehicle's interior space is provided as a reference point for the positioning or translation of the HMD.
[0016] In another improvement, the application is designed to compensate for HMD translations provided by the 6DOF algorithm, based on the determined computer vision-based HMD translation. This offers the advantage of correcting the erroneous positioning of at least one HMD within the vehicle's interior space, determined by the 6DOF algorithm, by repositioning it based on an edge model. Thus, a mobile AR / VR-HMD is used during dynamic driving in a dynamic environment. The system avoids the need for in-situ tracking when using VR and AR head-mounted displays without external trackers.
[0017] In one design, the application is designed to create an edge model based on images from at least one camera recorded by at least one camera. The system uses at least one camera to construct an edge model of the vehicle's interior space. This application is typically designed as a VR / AR application.
[0018] In one improved embodiment, the system is designed to conceal the environment perceived by at least one camera of at least one HMD. For example, the system is designed to disregard camera images in the calculation of creating the edge model, based on distance or spacing adjustments.
[0019] In an alternative design, the vehicle's adjustment device includes an edge model, which is designed to transmit the edge model to the HMD's application upon initial connection with the HMD. The application is designed to perform computer vision-based HMD translation calculations based on the edge model. Therefore, the alternative design is designed to replace the dynamic creation of the edge model of the interior space, transmitting information or the edge model to the HMD. Typically, the adjustment device includes at least one air interface designed to transmit the edge model to the HMD's air interface. Here, the edge model is typically stored in the corresponding vehicle's vehicle model information.
[0020] In one improved embodiment, the control unit is designed to transmit vehicle sensor data provided by the adjustment device via at least one air interface of the HMD to an application within the HMD for calculating the vehicle's self-motion. The application is designed to determine the vehicle's self-motion and, based on this self-motion, perform a computer vision-based translation calculation relative to the vehicle's interior space. This provides the advantage that the vehicle's translation can be taken into account to compensate for the HMD position calculated by the 6DOF algorithm.
[0021] Vehicle sensor data is typically detected by at least two sensors designed to transmit vehicle sensor data to at least one conditioning device, which is designed to store the vehicle sensor data. Typically, the vehicle sensor data is provided to at least one conditioning device with at least one air interface via at least one communication channel in the vehicle (e.g., via Flexray, CAN, or Ethernet). The conditioning device is designed to transmit the vehicle sensor data to the air interface of the HMD (Hardware Management Device) via the air interface.
[0022] In another improved embodiment, at least one air interface of the regulating device and / or HMD is a Bluetooth Low Energy (BLE) connection. In an alternative design, at least one air interface of the regulating device and / or HMD is a local Wi-Fi connection or a classic Bluetooth connection.
[0023] Optionally, at least one HMD includes at least one inertial measurement (IMU) unit, which typically includes at least one acceleration sensor and at least one rotation speed sensor, and is designed to detect sensor data. Thus, the IMU unit forms the sensing and measurement unit of the inertial navigation system.
[0024] Furthermore, the present invention relates to a method for 6DOF tracking of a mobile HMD in a vehicle during dynamic driving, including the aforementioned system.
[0025] According to the present invention, in a first step a, an edge model of the vehicle's interior space is provided. In a further step b, automated 6DOF tracking based on computer vision is provided. Here, steps a and b can be performed sequentially or simultaneously. In a further step c, HMD translation relative to the vehicle's interior space, based on computer vision, is calculated based on the edge model. In a further step d, HMD 11 translation provided by interior / exterior tracking based on a six-degrees-of-freedom (6DOF) algorithm is provided. Here, steps c and d can be performed sequentially or simultaneously. In a further step e, the HMD 11 translation based on the 6DOF algorithm is compensated by means of the computer vision-based HMD 11 translation.
[0026] In an improved version of this method, an edge model is created and provided based on at least one camera image recorded by at least one camera of the HMD. Therefore, the system is designed to take into account the camera images detected by at least one camera (which are also used for interior and exterior tracking) as the basis for creating an edge model of the vehicle's interior space. The resulting edge model can thus be used as a zero reference for HMD translation.
[0027] In an alternative improvement, the edge model is transmitted to the HMD by the vehicle's adjustment equipment upon initial connection between the vehicle and the HMD. Therefore, the alternative design is intended to dynamically create an edge model of the interior space, transmitting information or the edge model to the HMD. Here, the edge model is typically a portion of the vehicle model information stored in the vehicle's adjustment equipment. Attached Figure Description
[0028] The present invention is illustrated schematically with reference to the accompanying drawings, and is further described with reference to the drawings, wherein like parts are indicated by like reference numerals. Wherein:
[0029] Figure 1 An embodiment of the system according to the invention is shown, which has an edge model based on camera images from a camera integrated in an HMD.
[0030] Figure 2 It shows Figure 1 Another implementation of the system shown. Detailed Implementation
[0031] Figure 1 An embodiment of a system 10 according to the present invention is shown, the system 10 having an edge model based on camera images from a camera 15 integrated in an HMD 11. Here, the system 10 for detecting at least one mobile head-mounted display (HMD) 11 includes at least one vehicle 12 and at least one HMD 1.
[0032] The vehicle 12 includes at least one adjustment device 13 and at least one air interface 14, wherein the adjustment device 13 is designed to provide vehicle sensor data.
[0033] At least one HMD 11 includes at least one camera 15, a control unit 16, an air interface 18, and at least one application 17, wherein the control unit 16 of the HMD 11 is designed to perform in-situ tracking based on a 6DOF algorithm using at least one camera image recorded by at least one camera 15.
[0034] Application 17 is designed to provide an edge model of the interior space of vehicle 12 and to provide automated 6DOF tracking based on the edge model. Here, application 17 is designed to perform computer vision-based HMD translation calculations based on the edge model, wherein application 17 is designed to determine a computer vision-based HMD 11 translation relative to the interior space of vehicle 12. Furthermore, application 17 is designed to compensate for the HMD 11 translation provided by the 6DOF algorithm based on the determined computer vision-based HMD 11 translation.
[0035] Application 17 itself creates an edge model. Here, application 17 is designed to create an edge model based on at least one camera image recorded by at least one camera 15.
[0036] Typically, the adjustment device 13 of vehicle 12 is designed to transmit vehicle sensor data 19, determined by vehicle sensors of vehicle 12, to application 17, wherein application 17 is designed to determine the vehicle's self-motion based on the transmitted vehicle sensor data 19. System 10 is designed to calculate computer vision-based 6DOF HMD 11 tracking based on the determined self-motion of vehicle 12.
[0037] Figure 1 A method for 6DOF tracking of a mobile HMD 11 in a vehicle 12 during dynamic driving is also shown using the system 10 described above.
[0038] The method includes providing an edge model of the vehicle's interior space in a first step a. In a subsequent step b, automated 6DOF tracking based on computer vision is provided. Steps a and b can be performed selectively, sequentially, or simultaneously.
[0039] In another step c, a computer vision-based translation of the HMD 11 relative to the interior space of the vehicle 12 is calculated based on the edge model. In another step, the HMD 11 translation provided by interior-exterior tracking based on a six-degree-of-freedom (6DOF) algorithm is provided. Here, steps a and b can be performed sequentially or simultaneously. In another step e, the computer vision-based HMD 11 translation compensates for the HMD 11 translation provided by the 6DOF-based interior-exterior tracking.
[0040] In this embodiment of the method, an edge model is created and provided based on camera images recorded by at least one camera 15 of the HMD 11.
[0041] Here, at least one camera 15 is designed to perceive the environment and retain it in camera images. These camera images serve, in particular, as the basis for computer vision-based localization of at least one HMD 11. Based on these algorithms, an edge model is created from the camera images of the environment. Here, the spatial constraints or maximum distances used for evaluation are ensured to be considered only in the interior space of the vehicle 12 and not from a dynamic environment as a benchmark for creating the edge model.
[0042] Using the computed edge model, the X, Y, and Z positions of HMD 11 relative to the edge model can be calculated based on the algorithm. Here, the translation of HMD 11 relative to the edge model is typically used as the 0 reference for the desired HMD 11 camera position. In Application 17, the HMD 11 translation of the internal 6DOF tracking is corrected by transformation / reduction to the 0 reference, specifically by 3DOF tracking relative to the edge model.
[0043] Therefore, in the application, the actual position of the HMD in the vehicle is used to compensate for the camera position that is incorrectly calculated internally by the HMD due to the dynamic environment during vehicle movement. The mathematical operation for this purpose is the sum of the error vector and the Delta vector relative to 0 reference.
[0044] Figure 2 It shows Figure 1 Another embodiment of the system 10 shown is illustrated. The system 10 includes a vehicle 12 with an adjustment device 13 and an air interface 14, and an HMD 11 with a camera 15, a control unit 16, an air interface 18, and an application program 17. In this embodiment, the edge model is provided to the HMD 11 by the adjustment device 13 based on vehicle model information.
[0045] During the development of vehicle 12, various different vehicle 12 models exist. Typically, an edge model has been created from one of these models, and this edge model is stored in the adjustment device 13 along with other vehicle model information. In this embodiment, when HMD 11 is initially connected to vehicle 12, the edge model is transmitted to HMD 11 via air interface 14 of adjustment device 13.
[0046] Figure 2 It also shows in Figure 1 The method described in the accompanying drawings is for 6DOF tracking of a mobile HMD 11 in a vehicle 12 during dynamic driving using steps a to e. Here, application 17 is also designed to determine the self-motion of the vehicle 12 based on vehicle sensor data 19 transmitted by adjustment device 14, and to base further calculations on this self-motion. In this embodiment of the method, instead of determination based on camera images from at least one HMD, an edge model is transmitted by adjustment device 13 of the vehicle 12 to the HMD 11 when the vehicle 12 is initially connected to the HMD 11.
[0047] List of reference numerals in the attached diagram:
[0048] 10 System
[0049] 11 HMD
[0050] 12 vehicles
[0051] 13. Adjustment equipment
[0052] 14. Adjust the air interface of the equipment.
[0053] 15 cameras
[0054] 16 Control Unit
[0055] 17 Applications
[0056] 18 HMD's air interface
[0057] 19 Vehicle sensor data
Claims
1. A system (10) for detecting at least one mobile head-mounted display (HMD) (11), comprising at least one vehicle (12) and said at least one head-mounted display (11), wherein, The vehicle (12) includes at least one adjustment device (13) and at least one air interface (14), wherein the adjustment device (13) is designed to provide vehicle sensor data, and the at least one head-mounted display (11) includes at least one camera (15), a control unit (16), an air interface (18), and at least one application (17), wherein the control unit (16) of the head-mounted display (11) is designed to perform inward and outward tracking of the at least one mobile head-mounted display (11) based on a six-degrees-of-freedom (6DOF) algorithm using at least one camera image recorded by the at least one camera (15), and to determine the translation of the at least one mobile head-mounted display (11) relative to the interior space of the vehicle (12) based on a six-degrees-of-freedom algorithm, characterized in that the at least one mobile head-mounted display (11) includes at least one adjustment device (13) and at least one air interface (14), wherein the at least one head-mounted display (11) includes at least one camera (15), a control unit (16), an air interface (18), and at least one application (17 ...) and at least one air interface (18), wherein the at least one head-mounted display (11) includes at least one camera (15), a control unit (16) and at least one air interface (17), wherein the at least one head-mounted display (11) includes at least one camera (13) and at least one air interface (14), wherein the at least one head-mounted display (14) includes at least one camera (15), a control unit (16) and at least one air interface (17), wherein the at least one head-mounted display (15) includes at Application (17) is additionally designed to provide an edge model of the interior space of the vehicle (12) and to provide automated computer vision-based six-degree-of-freedom tracking of the at least one mobile head-mounted display (11) based on the edge model. The control unit (16) is designed to transmit vehicle sensor data provided by the adjustment device (13) via at least one air interface of the head-mounted display (11) to application (17) of the head-mounted display (11) for calculating the self-motion of the vehicle (12). The application (17) is designed to determine the self-motion of the vehicle (12) based on the vehicle sensor data and to perform computer vision-based translation of the at least one mobile head-mounted display (11) relative to the interior space of the vehicle (12) based on the self-motion.
2. The system (10) according to claim 1, characterized in that, The application (17) is designed based on computer vision-based six-degree-of-freedom tracking to enable computer vision-based head-mounted display (11) translation calculations to be based on an edge model, wherein the application (17) is designed to determine the computer vision-based head-mounted display (11) translation relative to the interior space of the vehicle (12).
3. The system (10) according to claim 1, characterized in that, The application (17) is designed to compensate for the translation of the head-mounted display (11) provided by a six-degree-of-freedom algorithm, based on the determined translation of the head-mounted display (11) based on the computer vision-based head-mounted display (11).
4. The system (10) according to any one of claims 1 to 3, characterized in that, The application (17) is designed to create an edge model based on at least one camera image recorded by at least one camera (15).
5. The system (10) according to any one of claims 1 to 3, characterized in that, The adjustment device (13) of the vehicle (12) includes an edge model, wherein the adjustment device (13) is designed to transmit the edge model to an application (17) of the head-mounted display (11) upon initial connection with the head-mounted display (11), wherein the application (17) is designed to perform computer vision-based translation calculations of the head-mounted display (11) based on the edge model.
6. The system (10) according to any one of claims 1-3, characterized in that, At least one air interface (14) of the adjustment device (13) and / or head-mounted display (11) is a Bluetooth Low Energy (BLE) connection.
7. A method for performing six-degree-of-freedom tracking of a mobile head-mounted display (11) in a vehicle (12) during dynamic driving using a system (10) according to any one of claims 1 to 6, comprising the steps of: a. Provide an edge model of the vehicle's interior space. b. Provides automated six-degree-of-freedom tracking based on computer vision. c. Calculate the translation of the head-mounted display (11) relative to the interior space of the vehicle (12) based on an edge model, using computer vision. d. Provides translation of the head-mounted display (11) by inward and outward tracking based on a six-degree-of-freedom (6DOF) algorithm. e. Compensate for translation of head-mounted display (11) based on in-and-out tracking by using translation of computer vision-based head-mounted display (11).
8. The method according to claim 7, wherein, An edge model is created and provided based on camera images recorded by at least one camera (15) of the head-mounted display (11).
9. The method according to claim 7, wherein, The edge model is transmitted to the head-mounted display (11) by the adjustment device (13) of the vehicle (12) when the vehicle (12) is first connected to the head-mounted display (11).