6 Degree of Freedom (6DOF) Tracking von mobilen Head Mounted Displays (HMD)

Abstract

A system (10) for detecting at least one mobile head-mounted display (HMD) (11), comprising at least one vehicle (12) and the at least one HMD (11), wherein the vehicle (12) comprises at least one control device (13) and at least one air interface (14), wherein the control device (13) is configured to provide vehicle sensor data, and wherein the at least one HMD (11) comprises firmware with a 6 Degree of Freedom (6DOF) algorithm integrated in the firmware, at least one camera (15), a control unit (16), an air interface (18), and at least one application (17), wherein the control unit (16) of the HMD (11) is configured to perform inside-out tracking based on the 6DOF algorithm with the aid of at least one camera image recorded by the at least one camera (15) and to determine a 6DOF algorithm-based translation of the at least one mobile HMD (11), characterized in thatthat the application (17) is additionally configured to provide an edge model of an interior of the vehicle (12) and, based on the edge model, to provide an automated computer vision-based 6DOF tracking of the at least one mobile HMD (11), based on the computer vision-based 6DOF tracking, to use the edge model as the basis for calculating a computer vision-based translation of the HMD (11), wherein the application (17) is configured to determine the computer vision-based translation of the HMD (11) relative to the interior of the vehicle (12), and, based on the determined computer vision-based translation of the HMD (11), to compensate for a translation of the HMD (11) provided by the 6DOF algorithm.

Description

The present invention relates to a system for detecting at least one mobile head mounted display (HMD), comprising at least one vehicle and the at least one HMD, wherein the vehicle comprises at least one control device and at least one air interface, wherein the control device is configured to provide vehicle sensor data, and wherein the at least one HMD comprises at least one camera, a control unit, an air interface and at least one application, wherein the control unit of the HMD is configured to carry out inside-out tracking based on a 6 degree of freedom (6OF) algorithm with the aid of at least one camera image recorded by the at least one camera and to determine a 6OF algorithm-based translation of the at least one mobile HMD.Virtual reality (VR) and augmented reality (AR) are technologies which are already widely used today for gaming and also productive applications. In this case, a user carries a so-called head mounted display (HMD). It is fundamentally important for the use of VR and AR headsets to precisely locate the HMD during use in order to be able to visually display the content correctly for the user. In particular, a 6OF translation in the X, Y, Z direction and a rotation about the aforementioned axes are determined.The localization or detection of an HMD is referred to as tracking. In this case, a distinction is made between inside-out tracking (from the outside of the HMD) and outside-in tracking (from the outside to the HMD). Previously, outside-in tracking for stationary application cases with external tracking sensors, for example via infrared, is state of the art.In the future, however, the development of HMD will also focus on mobile HMD, without external trackers, with inside-out tracking. However, present-day applications are limited to static environments.For example, a method for determining a pose of a camera on a head-mounted screen system in a vehicle is known from document EP 2 491 530 B1. The pose of the camera is determined optically in six degrees of freedom using an edge model from environmental data.From document US 2002 / 0 105 484 A1, a system for calibrating a display device worn on the head of a user for a representation of virtual objects in a real environment is known. The display device has a detection unit operating in six degrees of freedom for image data-based generation of a surroundings model.From document US 2010 / 0 109 976 A1, a device for tracking a display unit on the helmet of a vehicle occupant is known. The helmet has an optical sensor which serves with six degrees of freedom for detecting orientation marks of a vehicle interior and which is suitable for virtual reality applications.DE 10 2016 225 268 A1 discloses a system which comprises a vehicle and an HMD with a pose recognition unit. The HMD displays a VR / AR display content having a scope depending on a traveling state of the vehicle.DE 10 2014 225 222 A1 discloses a method for determining a position of an HMD relative to a head of a carrier of the HMD, which is carried out in particular for each new carrier after each renewed placement of the HMD or after each displacement of the HMD.DE 10 2013 210 746 A1 discloses an HMD for controlling a technical installation, in particular a vehicle. The HMD virtualizes real display and operating elements of the technical installation and always places the virtualized display and operating elements in such a way that they appear fixed to the technical installation depending on a determined pose of the HMD.The VR and AR-HMD known in the prior art implement tracking algorithms already in the firmware, wherein, however, a static environment is assumed. Due to the hardware-proximal implementation of the 6OF algorithms in the firmware in present-day HMD, it is not possible to deactivate translation of the inside-out tracking.It is an object of the present invention to provide a system that enables mobile HMD to be used even in dynamic environments.This object is achieved by a system having the features of claim 1. Advantageous refinements are the subject matter of the description and description of the figures.The present invention relates to a system for detecting at least one mobile head mounted display (HMD), comprising at least one vehicle and the at least one HMD, wherein the vehicle comprises at least one control device and at least one air interface, wherein the control device is configured to provide vehicle sensor data, and wherein the at least one HMD comprises at least one camera, a control unit, an air interface and at least one application, wherein the control unit of the HMD is configured to carry out inside-out tracking based on a 6 degree of freedom (6OF) algorithm and to determine a 6OF algorithm-based translation of the at least one mobile HMD with the aid of at least one camera image recorded by the at least one camera. The at least one HMD is generally designed as spectacles. Alternatively, the at least one HMD is designed as a lens. The 6OF algorithm is implemented integrated in a firmware of the HMD.According to the invention, the application is additionally configured to provide an edge model of an interior of the vehicle and, based on the edge model, to provide automated computer vision-based 6OF tracking of the at least one mobile HMD. Tracking is a localization or detection of the HMD.Computer vision means machine vision. The system is accordingly configured to enable machine vision into the interior of the vehicle by means of the at least one camera of the HMD. In addition, the system is configured to provide an edge model of the interior space.The system offers the advantage that, by including the edge model in the computer vision-based 6OF tracking, consideration of a dynamic environment, which is perceived by the at least one camera, for example, by window panes of the vehicle, is corrected. This prevents a mislocalization of the HMD in the vehicle, which may lead to a sudden change in a gaze position of a user within the VR and AR contents. The viewing position of the user is also the viewing position of the at least one HMD camera, which the computer vision provides based viewing.Furthermore, according to the invention, the application is set up based on the computer vision-based 6OF tracking to form the basis of the edge model of a calculation of a computer vision-based translation of the HMD, wherein the application is set up to determine the computer vision-based translation of the HMD relative to the interior of the vehicle. With the aid of the edge model of the interior of a respective vehicle, the system is set up to localize an exact position of the at least one HMD in the interior of the vehicle. In this case, the interior of the vehicle is provided as a reference point for the localization or translation of the HMD.Furthermore, according to the invention, the application is configured to compensate for a translation of the HMD provided by the 6OF algorithm on the basis of the determined computer vision-based translation of the HMD. This offers the advantage that an incorrectly localized position of the at least one HMD in the interior of the vehicle determined by the 6OF algorithm is corrected by the new position determination on the basis of the edge model. This allows mobile AR / VR HMD to be used during dynamic driving in a dynamic environment. The system prevents use of VR and AR HMD without external trackers with inside-out tracking.In one configuration, the application is configured to generate the edge model on the basis of the at least one camera image recorded by the at least one camera. The system is configured with the aid of the at least one camera to image the interior of the vehicle as an edge model. The application is generally set up as a VR / AR application.In one development, the system is configured to screen out an environment perceived by the at least one camera of the at least one HMD. For example, the system is configured to exclude camera images from consideration in the calculation of the creation of the edge model on the basis of a distance setting.In an alternative embodiment, the control device of the vehicle comprises an edge model, wherein the control device is configured to transmit the edge model to the application of the HMD upon a first connection to the HMD, wherein the application is configured to use the edge model as the basis for the computer vision-based calculation of the translation of the HMD. Thus, the alternative embodiment is configured to transmit information or the edge model to the HMD instead of a dynamic creation of the edge model of the interior. As a rule, the control device comprises at least one air interface which is configured to transmit the edge model to an air interface of the HMD. The edge model is generally stored in vehicle model information of a respective vehicle.In one development, the control unit is configured to transmit the vehicle sensor data provided by the control device to the application of the HMD via the at least one air interface of the HMD in order to calculate a self-movement of the vehicle, wherein the application is configured to determine the self-movement of the vehicle and to use the calculation of the computer vision-based translation relative to the interior of the vehicle as a basis. This provides the advantage that translation of the vehicle may be taken into account to compensate for the position of the HMD calculated by the 6OF algorithm.The vehicle sensor data are generally captured by at least two sensors which are configured to transmit the vehicle sensor data to at least one control device, wherein the control device is configured to store the vehicle sensor data. As a rule, the vehicle sensor data are provided via at least one communication channel in the vehicle, for example via Flexray, CAN or Ethernet, to the at least one control device with the at least one air interface. The control device is configured to transmit the vehicle sensor data to the air interface of the HMD via the air interface.In a further development, the at least one air interface of the control device and / or of the HMD is a Bluetooth Low Energy (BLE) connection. In an alternative embodiment, the at least one air interface of the control device and / or of the HMD is a local WIFI connection or a classic Bluetooth connection.Optionally, the at least one HMD comprises at least one inertial measurement (IMU) unit, which generally comprises at least one acceleration sensor and at least one rotation rate sensor, and is configured to acquire sensor data. Thus, the IMU unit is a sensory measurement unit of an inertial navigation system.Furthermore, the present invention relates to a method for performing 6OF tracking of a mobile HMD in a vehicle during dynamic driving, comprising a system described above.According to the invention, in a first step a, an edge model of an interior of the vehicle is provided. In a further step b, automated computer vision-based 6OF tracking is provided. Steps a and b can be carried out successively or simultaneously. In a further step c, a computer vision-based translation of the HMD relative to the interior of the vehicle is calculated based on the edge model. In a further step d, translation of the HMD 11 provided by an inside-out tracking based on a 6 degree of freedom (6OF) algorithm is provided. Steps c and d can be carried out one after the other or simultaneously. In a further step e, the translation of the HMD 11 based by the 6OF algorithm is compensated with the aid of the computer vision-based translation of the HMD 11.In a development of the method, the edge model is created and provided on the basis of at least one camera image recorded by at least one camera of the HMD. The system is thus configured to use, based on the camera images captured by the at least one camera, which are also used for inside-out tracking, as a basis for creating an edge model of the vehicle interior. The edge model thus created is usable as a 0 reference for the translation of the HMD.In an alternative development, the edge model is transmitted to the HMD by a control device of the vehicle when the vehicle is connected to the HMD for the first time. Thus, the alternative embodiment is configured to transmit information or the edge model to the HMD instead of a dynamic creation of the edge model of the interior. The edge model is generally part of vehicle model information which is stored in a control device of the vehicle.The invention is schematically illustrated in the drawing by means of embodiments and is further described with reference to the drawing, wherein the same components are denoted by the same reference numerals. It shows: FIG. 1 shows an embodiment of a system according to the invention with an edge model based on camera images of cameras integrated in the HMD, FIG. 2 shows a further embodiment of the system shown in FIG. 1.FIG. 1 shows an embodiment of a system 10 according to the invention having an edge model based on camera images of cameras 15 integrated in an HMD 11. The system 10 for capturing at least one mobile head mounted display (HMD) 11 comprises at least one vehicle 12 and the at least one HMD 1.The vehicle 12 comprises at least one control device 13 and at least one air interface 14, wherein the control device 13 is configured to provide vehicle sensor data.The at least one HMD 11 comprises at least one camera 15, a control unit 16, an air interface 18 and at least one application 17, wherein the control unit 16 of the HMD 11 is configured to carry out inside-out tracking based on a 6OF algorithm with the aid of at least one camera image recorded by the at least one camera 15 and to determine a 6OF algorithm-based translation of the HMD 11.The application 17 is configured to provide an edge model of an interior of the vehicle 12 and to provide automated computer vision-based 6OF tracking based on the edge model.Application 17 is configured to form the basis for the edge model of a calculation of a computer vision-based translation of the HMD, application 17 being configured to ascertain the computer vision-based translation of HMD 11 relative to the interior of vehicle 12. In addition, the application 17 is configured to compensate for a translation of the HMD 11 provided by the 6OF algorithm on the basis of the determined computer vision-based translation of the HMD 11.The application 17 generates the edge model itself. The application 17 is configured to generate the edge model based on the at least one camera image recorded by the at least one camera 15.In general, the control device 13 of the vehicle 12 is configured to transmit vehicle sensor data 19 ascertained based on vehicle sensors of the vehicle 12 to the application 17, wherein the application 17 is configured to ascertain an own movement of the vehicle based on the transmitted vehicle sensor data 19.The system 10 is configured to use the determined own movement of the vehicle 12 as a basis for the calculation of the computer vision-based 6OF tracking of the HMD 11.FIG. 1 also shows the method for carrying out 6OF tracking of a mobile HMD 11 in the vehicle 12 during dynamic travel using the system 10 described above.In a first step a, the method comprises providing an edge model of an interior of the vehicle. In a further step b, automated computer vision-based 6OF tracking is provided. Steps a and b can be carried out optionally one after the other or simultaneously.In a further step c, a computer vision-based translation of the HMD 11 relative to the interior of the vehicle 12 is calculated based on the edge model. In a further step, an inside-out tracking provided by a 6 degree of freedom (6OF) algorithm is provided for translation of the HMD 11. Steps a and b can be carried out optionally one after the other or simultaneously. In a further step e, the translation of the HMD 11 provided by a 6OF algorithm is compensated for with the aid of the computer-vision-based translation of the HMD 11.In the present embodiment of the method, the edge model is created and provided based on a camera image recorded by at least one camera 15 of the HMD 11.In this case, the at least one camera 15 is configured to perceive the environment and to capture it in camera images. These camera images are used in particular as a basis for computer vision-based localizations of the at least one HMD 11. Based on the algorithms, an edge model is created from the camera images of the environment. In this case, a spatial limitation or maximum distance for the evaluation can ensure that only references in the interior of the vehicle 12 and not from the dynamic environment are used for the creation of the edge model.With the aid of the calculated edge model, an X, Y and Z position of the HMD 11 relative to the edge model can be calculated on the basis of algorithms. The translation of the HMD 11 relative to the edge model is generally used as a 0-reference of the desired HMD 11 camera position. In the application 17, the translation of the HMD 11 internal 6D tracking is corrected by a transformation to the 0-reference, in particular a 3OF tracking relative to the edge model.Thus, the position of the HMD camera in the application that is internally incorrectly calculated by the dynamic environment during the travel of the vehicle is compensated by the actual position of the HMD in the vehicle. The mathematical operation for this is a vector addition of an erroneous vector plus a delta vector to the 0 reference.FIG. 2 shows a further embodiment of the system 10 shown in FIG. 1, which shows the system 10 comprising the vehicle 12, the control device 13 and the air interface 14, and the HMD 11 having the camera 15, the control unit 16, the air interface 18 and the application 17.During the development process of the vehicle 12, there are a plurality of different models of the vehicle 12, and an edge model has generally been created from one of these models, which edge model is stored together with further vehicle model information in the control device 13. In the present embodiment, when the HMD 11 is first connected to the vehicle 12, the edge model is transmitted to the HMD 11 via the air interface 14 of the control device 13.FIG. 2 also shows the method described in the description of the figures for FIG. 1 for carrying out the 6OF tracking of the mobile HMD 11 in the vehicle 12 during a dynamic trip, having steps a to e. Application 17 is likewise configured to ascertain an own movement of vehicle 12 on the basis of vehicle sensor data 19 transmitted by control device 14 and to use this as a basis for the further calculation. In the present embodiment of the method, however, instead of a determination based on camera images of the at least one HMD, the edge model is transmitted by a control device 13 of the vehicle 12 to the HMD 11 upon a first connection of the vehicle 12 to the HMD 11.List of reference characters10 System 11 HMD 12 Vehicle 13 Control device 14 Air interface Control device 15 Camera 16 Control unit 17 Application 18 Air interface HMD 19 Vehicle sensor data

Claims

System (10) for detecting at least one mobile head mounted display (HMD) (11), comprising at least one vehicle (12) and the at least one HMD (11), wherein the vehicle (12) comprises at least one control device (13) and at least one air interface (14), wherein the control device (13) is configured to provide vehicle sensor data, and wherein the at least one HMD (11) comprises firmware with a 6 degree of freedom (6OF) algorithm integrated in the firmware, at least one camera (15), a control unit (16), an air interface (18) and at least one application (17), wherein the control unit (16) of the HMD (11) is configured to receive a signal from the vehicle, Inside-out tracking based on the 6OF algorithm and to determine a 6OF algorithm-based translation of the at least one mobile HMD (11), characterized in that the application (17) is additionally configured to provide an edge model of an interior of the vehicle (12) and to provide automated computer vision-based 6D tracking of the at least one mobile HMD (11) based on the computer vision-based 6D tracking based on the computer vision-based 6D tracking to form the edge model as a basis for a calculation of a computer vision-based translation of the HMD (11), wherein the application (17) is configured to:, determining the computer vision-based translation of the HMD ( 11) relative to the interior of the vehicle ( 12), and compensating a translation of the HMD ( 11) provided by the 6OF algorithm based on the determined computer vision-based translation of the HMD ( 11).The system (10) according to claim 1, characterized in that the application (10) is configured to generate the edge model based on the at least one camera image recorded by the at least one camera (15).System (10) according to Claim 1 or 2, characterized in that the control device (13) of the vehicle (12) comprises an edge model, wherein the control device (13) is set up to transmit the edge model to the application (17) of the HMD (11) on a first connection to the HMD (11), wherein the application (17) is set up to form the basis of the edge model of the computer vision-based calculation of the translation of the HMD (11).System (10) according to Claim 1, characterized in that the control unit (16) is set up to transmit the vehicle sensor data provided by the regulating device (13) to the application (17) of the HMD (11) via the at least one air interface of the HMD (11), the application (17) being set up to determine the own movement of the vehicle (12) on the basis of the vehicle sensor data and to use the calculation of the translation based on computer vision relative to the interior of the vehicle (12).System (10) according to one of the preceding claims, characterized in that the at least one air interface (14) of the control device (13) and / or of the HMD (11) is a Bluetooth Low Energy (BLE) connection.Method for performing 6OF tracking of a mobile HMD (11) in a vehicle during a dynamic trip (12), with a system (10) according to one of Claims 1 to 5, comprising the steps: a. providing an edge model of an interior of the vehicle, b. providing automated computer vision-based 6D tracking, c. calculating a computer vision-based translation of the HMD (11) relative to the interior of the vehicle (12) based on the edge model, i.e. providing an inside-out tracking provided by a 6 Degree of Freedom (6OF) algorithm, Compensation for inside-out tracking based translation of the HMD ( 11) with the aid of computer vision based translation of the HMD ( 11).Method according to claim 6, wherein the edge model is created and provided based on a camera image recorded by at least one camera (15) of the HMD (11).Method according to Claim 7, wherein the edge model is transmitted to the HMD (11) by a control device (13) of the vehicle (12) when the vehicle (12) is connected to the HMD (11) for the first time.

Images