Method and device for determining an installation position of a vehicle-mounted inertial sensor in a motor vehicle

Description

Technical field

[0001] The invention relates to motor vehicles with built-in or vehicle-mounted inertial sensors, in particular for supporting the tracking of a mobile device, such as data glasses. The invention further relates to measures for determining the installation pose of the vehicle-mounted inertial sensors based on a tracking system for detecting a pose of the mobile device and an inertial sensor of the mobile device. Technical background

[0002] Data glasses, also called head-mounted displays, are known. These devices, with the aid of a display device, can display an image on one or two display surfaces in the wearer's field of vision. The display surfaces can correspond to reflective surfaces that direct images into the wearer's eyes. The viewing openings of the data glasses are transparent, allowing the real environment to be perceived through the data glasses in the usual way. The display surfaces are located in the viewing openings, allowing information to be displayed, such as text, symbols, graphics, video displays, and the like, to be displayed overlaid on the perception of the environment.

[0003] The information can usually be presented to the wearer of the data glasses in a contact-analog manner, i.e. it can be presented in such a way that it is superimposed as object information on a specific assigned object in the real environment or is oriented towards it, or that the object information to be displayed is displayed in a specific orientation of the data glasses or their wearer.

[0004] Furthermore, the contact-analog object information can be presented in such a way that it appears perspectively correct with respect to the object in the real environment, ie the illusion is created that the object in the real environment has actually been supplemented by the additional feature of the visual object information.

[0005] However, in order to display the object information on the display surfaces of the smart glasses in a contact-analog manner, it is necessary to know the position of the object in the environment and the user's gaze direction. When wearing the smart glasses, the user's gaze direction is permanently assigned to the glasses' pose, i.e., the 3D position and the 3D orientation of the smart glasses.

[0006] To ensure the lowest possible latency and rapid pose updates, the pose of the smart glasses is determined using an internal pose recognition unit. The pose recognition unit typically comprises an inertial sensor and a computing device, e.g., a microprocessor. The inertial sensor can capture movements in the form of accelerations, which can be converted into a current pose of the smart glasses through integration. Such pose recognition units integrated into the smart glasses exhibit an increasing pose recognition error due to the integration process. This error can be corrected using pose recognition from a tracking system external to the smart glasses.This external pose recognition can, for example, be camera-based, in particular using an interior camera that is directed at the head of the wearer of the data glasses in order to recognize a glasses pose of the data glasses with respect to a vehicle coordinate system.

[0007] The absolute pose of the data glasses, which is used to correct the internal detection of the data glasses, can be determined by a tracking system external to the glasses but fixed to the vehicle. For example, an interior camera detects the head of the data glasses wearer and, by evaluating the camera images, either determines the head pose and derives the data glasses' pose from this, or directly detects and determines the data glasses' pose. One difficulty with these so-called outside-in tracking systems is transmitting the absolute pose information determined outside the data glasses to the data glasses with a sufficiently low latency, particularly over a wireless communication connection, so that the data glasses can output contact-analog representations without delay or with a non-disruptive delay.

[0008] Another tracking method can provide distributed tracking for smart glasses. This requires a pair of inertial sensors in the smart glasses as well as a vehicle inertial sensor installed in the vehicle or mounted on the vehicle. In addition to improving the tracking of smart glasses, vehicle inertial sensors can also be necessary to improve the localization accuracy of the vehicle.

[0009] The exact knowledge of the installation position / pose of the vehicle inertial sensors relative to the vehicle coordinate system is necessary both for the distributed tracking of data glasses and for supporting vehicle localization.

[0010] Typically, the installation orientation or position of the vehicle's inertial sensors is either derived from CAD data or determined by manual calibration. For inertial sensors not permanently installed in the vehicle, the approximate orientation and pose can be specified by a structural design, such as a mobile device located in a predefined fixture.

[0011] The document DE 10 2018 201 509 A1 discloses a method for operating a display system with data glasses provided in a motor vehicle, comprising the following steps: detecting glasses movement information with a glasses movement indication and a timestamp corresponding to a time of detecting the relative movement of the data glasses with respect to the vehicle environment indicated by the glasses movement indication; by the data glasses; Receiving glasses pose information with a glasses pose specification and a timestamp corresponding to a time of detecting the absolute pose of the data glasses specified by the glasses pose specification in relation to a vehicle coordinate system; receiving vehicle movement information with a vehicle movement specification and a timestamp corresponding to a time of detecting the relative movement of the motor vehicle with respect to the vehicle environment specified by the vehicle movement specification; determining a current glasses pose specification in the data glasses depending on the glasses pose information, the vehicle movement information, and the glasses movement information; and operating the display system depending on the current glasses pose specification.

[0012] The document DE 10 2019 103 360 A1 discloses a method for operating a display system with an assistance system and data glasses, in particular in a motor vehicle, comprising the following steps: providing a glasses pose specification that specifies a glasses pose of at least one display surface of the data glasses; providing an eye pose specification that specifies an eye pose of at least one eye of the wearer of the data glasses; determining calibration information that specifies a relative offset of the at least one display surface of the data glasses to the at least one eye of the wearer of the data glasses; and using the calibration information to display a virtual object on the at least one display surface of the data glasses.

[0013] It is an object of the present invention to provide a method for determining the installation position of an inertial sensor system permanently installed in the vehicle, which method also allows it to be used as part of a tracking method for a terminal device movable in the vehicle, such as data glasses. Disclosure of the invention

[0014] This object is achieved by the method for determining an installation pose of an inertial sensor mounted on a vehicle in a vehicle for use in a display system with a mobile terminal, in particular data glasses, according to claim 1 and a display system according to the independent claims.

[0015] Further embodiments are specified in the dependent claims.

[0016] According to a first aspect, a method for determining an installation pose of a vehicle inertial sensor system in a vehicle with data glasses that can be moved in the vehicle is provided, comprising the following steps: Determining a glasses pose of the data glasses using a vehicle-mounted tracking system in order to obtain an absolute glasses pose in a vehicle coordinate system at different times; detecting an acceleration value of the vehicle by the vehicle inertial sensor; detecting a glasses acceleration value by a glasses inertial sensor in the data glasses; determining the installation pose of the vehicle inertial sensor depending on the absolute glasses poses, the acceleration value of the data glasses and the acceleration value of the vehicle.

[0017] In particular, the installation pose of the vehicle inertial sensor system can be used to provide a calibration of the installation pose of the vehicle inertial sensor system and / or to improve tracking of the data glasses or localization of the vehicle.

[0018] Furthermore, the acceleration information of the data glasses and / or the acceleration information of the vehicle can each comprise one to three translational accelerations and / or one to three rotational accelerations with respect to at least one spatial direction.

[0019] The above method is based on a display system with data glasses in a vehicle, where tracking is supported by a tracking system permanently installed in the vehicle. The tracking system can, for example, provide an interior camera aimed at the driver along with camera image analysis or another localization device. The tracking system has a precisely known installation pose, which can, for example, be determined from CAD data or can be measured or adjusted during production. The tracking system with the interior camera can directly determine a glasses pose in a known manner by detecting the driver wearing the data glasses or the data glasses. The glasses pose is based on the known installation pose of the interior camera with respect to the vehicle coordinate system.The vehicle coordinate system is firmly oriented towards the vehicle body and thus has a firmly defined and predetermined reference to the installation pose of the tracking system or the interior camera.

[0020] The data glasses are equipped with an inertial sensor to enable their own tracking relative to the vehicle's coordinate system. The inertial sensor acquires translational and / or rotational acceleration data in one or more spatial directions. An analysis performed within the data glasses allows tracking of the data glasses at a very high frequency. However, due to the relative measurement of the inertial sensor, a pose change is determined by integrating acceleration values twice, which leads to an increasing error. The pose change is determined relative to a specific point in time at which an absolute pose of the glasses is known based on the analysis of the vehicle's internal tracking system. The current pose of the glasses can be determined from the pose change and the absolute pose of the glasses.

[0021] A fixed vehicle inertial sensor system can also be installed in the vehicle to provide an additional correction option for the glasses tracking. For example, the data glasses can also be tracked by comparing the acceleration data from the data glasses with that of the vehicle. The vehicle inertial sensor system can also be used to improve the localization of the vehicle.

[0022] Thus, the pose of the data glasses can be determined using both the vehicle-mounted tracking system and the vehicle inertial sensor. A key prerequisite for this, however, is that the interior camera and the vehicle inertial sensors have a fixed reference to the vehicle coordinate system. While this is easily and conventionally achieved for the interior camera due to the installation situation, vehicle inertial sensors are often provided in electronic modules that cannot be arranged in the vehicle in a stable and robust manner with a fixed reference to the vehicle coordinate system.

[0023] The above method now provides for determining the correct actual installation pose of the vehicle inertial sensors based on the absolute eyeglass pose determined using the tracking system, the acceleration data from the eyeglass inertial sensors, and the acceleration data from the vehicle inertial sensors. In particular, a deviation in the orientation of the vehicle inertial sensors from a reference installation pose defined for the vehicle coordinate system is determined and provided as calibration information. This calibration information represents offsets of the installation position and installation orientation.

[0024] The determination is carried out by evaluating the difference between two absolute glasses poses recorded consecutively by the tracking system and a double temporal integration of the acceleration data of the data glasses and the vehicle recorded for this period, in particular in the case of a relative movement of the data glasses to the vehicle coordinate system during the recording period.

[0025] Furthermore, the installation pose of the vehicle inertial sensor can be determined depending on a minimization of an error function of an acceleration balance from the pose change of the absolute glasses poses, the acceleration information recorded by the glasses inertial sensor and the acceleration information recorded by the vehicle inertial sensor.

[0026] According to a further aspect, a display system for determining an installation pose of a vehicle inertial sensor system in a vehicle with data glasses that can be moved in the vehicle is provided, wherein the display system is designed to: Determining a glasses pose of the data glasses using a vehicle-mounted tracking system in order to obtain an absolute glasses pose in a vehicle coordinate system at different times; detecting an acceleration value of the vehicle by the vehicle inertial sensor; detecting a glasses acceleration value by a glasses inertial sensor in the data glasses; determining the installation pose of the vehicle inertial sensor depending on the absolute glasses poses, the acceleration value of the data glasses and the acceleration value of the vehicle. Short description of the drawing

[0027] Embodiments are explained in more detail below with reference to the attached drawings. They show: Figure 1 shows a schematic representation of a display system with data glasses for use in a motor vehicle; and Figure 2 shows a flowchart illustrating a method for determining an installation pose of a vehicle inertial sensor system. Description of embodiments

[0028] Figure 1shows a schematic representation of a display system 1, in particular for use in a motor vehicle as a technical system. The display system 1 comprises a stationary assistance system 2, which has a communication connection 4 with data glasses 3. The communication connection 4 is designed as a data transmission channel, e.g. in the form of a wireless communication connection or a wired communication connection. The communication connection 4 is capable of transmitting any type of data and information between the assistance system 2 and the data glasses 3, for example based on packet-based data transmission. The communication connection 4 can be based, for example, on WiFi, Bluetooth, Bluetooth Low Energy or a comparable standardized radio protocol.

[0029] The data glasses 3 comprise two transparent lenses 32, which are enclosed in a frame 31 in a manner known per se. The frame 31 is provided with temples 33 so that the data glasses 3 can be worn on the head of a user in a manner known per se.

[0030] One or both viewing panes 32 (lenses) are further provided with a transparent display surface 35, through which a suitable device, such as a display device 36 arranged on the frame 31, can project a display image for displaying virtual information objects into the eye of the wearer of the data glasses 3. The display device 36 can comprise a microprocessor or a comparable computing unit and a display unit, such as a projection device or the like. The display unit can be configured to direct the electronically generated display image onto the display surface 35 and to project / display it there.

[0031] Due to the transparent design of the display surface 35, the electronically generated image can be superimposed on the real environment perceivable through the display surface 35. Using the display device 36, a virtual information object, such as text, a symbol, video information, a graphic, or the like, can be displayed on one or both display surfaces 35.

[0032] The data glasses 3 can be worn on the user's head like a typical visual aid, with the frame 31 of the data glasses 3 resting on the user's nose and the temples 33 resting laterally against the user's head. The user's straight-ahead viewing direction is then determined by the transparent display surfaces 35 of the viewing windows 32, so that the user's viewing direction, which is predetermined by an eye position and an optical viewing axis (eye axis), has a fixed reference to the orientation of the data glasses 3. This reference depends individually on the wearer of the data glasses 3 and is specified by calibration information.

[0033] To display information objects, corresponding object information in the form of object data is transmitted from the assistance system 2 to the data glasses 3. The object data indicates the type of information object, such as a text object, an icon or other identification of a display area, the viewing angle range(s) in which the information object is to be displayed on the display surface, and the object position on the display surface at which the information object is to be displayed.

[0034] To detect the pose of the data glasses 3, a glasses inertial sensor system 38 can be provided, which is designed, for example, in the form of a 6-DoF inertial sensor. This sensor typically has three gyroscopes and three acceleration sensors for all spatial directions. This sensor provides a glasses movement indication in the form of one to three translational and / or one to three rotational acceleration indications. The glasses movement indication initially refers to a glasses coordinate system that is specified by the installation pose of the glasses inertial sensor system 38.

[0035] With the help of a control unit 37, object information is received and processed by the assistance system 2 via a communication device 39, so that it is displayed in the respective viewing angle range into which the user of the data glasses 3 is looking. Furthermore, the control unit 37 uses the glasses' movement information to determine a current pose of the data glasses 3 by integrating the movement information.

[0036] The assistance system 2 can be part of a vehicle assistance system and, in particular, can be provided in a fixed location in the motor vehicle. The assistance system 2 can be equipped with a communication unit 21 that enables the communication connection 4 between the data glasses 3 and the assistance system 2.

[0037] The assistance system 2 can further be equipped with a vehicle inertial sensor system 24 that is permanently installed in the vehicle or fixedly mounted in the vehicle interior, such as a smartphone in a corresponding position. The vehicle inertial sensor system 24 detects translational accelerations of the vehicle in translational spatial directions as well as rotational accelerations in rotational spatial directions with respect to the vehicle coordinate system.

[0038] The assistance system 2 can further be connected to an environment detection system 25, which has one or more cameras. The environment detection system 25 can detect the surroundings of the motor vehicle. The one or more cameras can include, for example, an RGB, IR, fisheye camera, a dynamic vision sensor, and the like.

[0039] The assistance system 2 can have a processor unit 23, by which at least one virtual information object for display in the data glasses 3 is generated and / or determined in a conventional manner depending on a geographical position of the motor vehicle and depending on an image of the environment captured by the environment detection system 25. The position of the at least one virtual information object is specified with respect to a vehicle-fixed coordinate system. The position of the virtual information object can be assigned to a viewing angle segment. For display in the data glasses 3, object information is generated which specifies the viewing angle segment, object content, and object type of the display of the at least one virtual information object in the data glasses 3.

[0040] The viewing angle segment comprises an angular range in the direction of head rotation (around the vertical axis) and in the direction of head tilt (around the transverse axis of the motor vehicle), in which the user, i.e. the wearer of the data glasses 3, is to be shown information objects fixed to the glasses, i.e. at a fixed position on a display surface of the data glasses 3

[0041] An interior camera system can be provided as the image-capturing device of a tracking system 26, which can be equipped, for example, with one or more interior cameras to capture the camera-based glasses pose of the data glasses 3. For this purpose, the one or more interior cameras are aligned with the wearer of the data glasses 3. Furthermore, the installation positions and orientations of the one or more interior cameras in the vehicle are precisely known. The processor unit 23 is designed to capture the recordings of the interior camera system and, using methods known per se, to determine the camera-based glasses pose from the recordings. The information about the camera-based glasses pose of the data glasses 3 is thus available in the assistance system 2. The camera-based glasses pose is determined by the interior camera system with reference to the vehicle coordinate system as a reference coordinate system, which is permanently assigned to the vehicle.

[0042] Furthermore, the assistance system 2 has a vehicle-mounted inertial sensor system 24 for detecting movement information of the vehicle with respect to its movement in the environment. The movement information includes translational and / or rotational acceleration information, each in / with respect to up to three spatial directions. The installation position and orientation of the vehicle-mounted inertial sensor system 24 are specified according to a reference installation pose.

[0043] Although the installation position and orientation of the vehicle-mounted inertial sensor system is usually known, translational or rotational deviations of the installation pose from the reference installation pose can occur due to manufacturing tolerances.

[0044] Using the flow chart of the Figure 2 the method for determining the installation pose of the vehicle inertial sensor system 24 is described in more detail.

[0045] In step S1, the glasses pose is determined using the tracking system 26 as the imaging device. For this purpose, the interior camera of the tracking system 26 detects the driver of the vehicle wearing the data glasses 3 and, by evaluating the resulting camera images, determines the absolute glasses poses of the data glasses 3 with respect to the vehicle coordinate system at a specific point in time. The evaluation is performed using known image evaluation methods. The glasses pose can be determined, for example, using a feature-based method by tracking prominent points in the image and determining the glasses pose from this. In addition, a learning algorithm in the form of an appropriately trained neural network can also be used to detect the glasses pose.

[0046] In step S2, acceleration data, i.e., 6DoF acceleration data with respect to all spatial directions, is acquired in the vehicle using the vehicle-mounted inertial sensor 24. For this purpose, the acceleration data are recorded at a high acquisition frequency and buffered.

[0047] In step S3, acceleration data from the data glasses 3, i.e. 6DoF acceleration data with respect to all spatial directions, are determined using the glasses inertial sensor system 38.

[0048] In step S4, the acquired information, i.e. the absolute glasses poses acquired by the tracking system, the acceleration data of the data glasses 3 and the vehicle acquired in the data glasses 3 and the vehicle, are combined, either in the assistance system 2 or in the data glasses 3, so that the acceleration data of the vehicle inertial sensor system 24, the acceleration data of the glasses inertial sensor system 38 and the absolute glasses pose determined by the tracking system 26 are provided in a time-synchronized manner.

[0049] When the data glasses 3 move, the corresponding acceleration data and the absolute glasses poses can now be recorded. By comparing the recorded data, any error is detected, which can be used to correct the installation pose of the vehicle inertial sensor system 24. For example, a movement of the data glasses 3 with respect to the vehicle coordinate system can be determined by calculating the difference between the acceleration data a glasses of the glasses inertial sensor system 38 and the acceleration data a vehicle of the vehicle inertial sensor system 24, which can be compared with the absolute glasses pose P glasses determined successively at different times via the tracking system 26.

[0050] For example, an installation pose of the tracking system 26 refers to the vehicle coordinate system, so that the absolute glasses poses P are determined with respect to the vehicle coordinate system.

[0051] The alignment of the tracking system 26 with the measured values of the glasses inertial sensor 38 and the vehicle inertial sensor 24 takes place by minimizing an error function. The error error position for the position is obtained by comparing the glasses acceleration data from the tracking system 26: is determined by the double time derivative of the pose P (position and orientation separately) with the measured acceleration values of the inertial sensors and a pair of glasses and a vehicle . The same applies analogously to the orientation with the angular velocities. The movement of the vehicle results in additional acceleration quantities for the formulation of the position error. term acc,tangential , term acc,coriolis and term acc,centripetal . These include, among other things, the installation position of the vehicle inertial sensors. fehler position = R <mprescripts / > IMF , f IMU , b ⋅ a brille − a fahrzeug − d 2 dt 2 P t 2 − P t 1 - term acc,tangential - term acc,coriolis - term acc,centripetal where only the position of the pose P is taken into account and fehler orientierung = d 2 dt 2 P t 2 − P t 1 − ω brille − R <mprescripts / > IMF , b IMU , f ⋅ ω fahrzeug where only the orientation of the pose P is taken into account.

[0052] The tangential acceleration term acc,tangential is obtained by multiplying the vehicle's rotational acceleration ω̇ vehicle with the position of the glasses in the vehicle. The Coriolis acceleration term acc,coriolis is calculated with 2 · ω vehicle · v glasses, where v glasses describes the speed of the glasses in the vehicle (determined by simple time derivative of the glasses position). The centripetal acceleration term acc,centripetal is calculated with ω vehicle · ω vehicle · p where p the position of the data glasses in the vehicle. The term R <mprescripts / > IMF , f IMU , b describes the rotation from the glasses inertial sensor 38 to the vehicle inertial sensor 24.

[0053] The total error results from the sum of orientation error and position error: fehler gesamt = fehler position + fehler orientierung

[0054] By adjusting or minimizing the total error, the installation pose of the vehicle inertial sensor 24 can be determined. The size of the error function to be changed is the installation position of the vehicle inertial sensor 24. min pose , imu , fahrzeug fehler gesamt

[0055] In step S5, the determined installation pose of the vehicle inertial sensor system 24 is used to improve the tracking of the data glasses 3 and / or the localization of the vehicle. List of reference symbols

[0056] 1Display system 2Assistance system 3Data glasses 4Communication connection 21Communication unit 23Processor unit 24Vehicle inertial sensors 25Environment detection system 26Tracking system 31Frame 32Viewing windows 33Temples 35Display surface 36Display device 37Control unit 38Glasses inertial sensors 39Communication device

Claims

1. Method for determining an installation pose of a vehicle inertial sensor system (38) in a vehicle using smartglasses (3) that are movable in the vehicle, comprising the following steps: - determining (S1) a glasses pose of the smartglasses (3) using a vehicle-fixed tracking system in order to obtain an absolute glasses pose (P) in a vehicle coordinate system at various times; - detecting (S2) acceleration information of the vehicle externally to the glasses by way of the vehicle inertial sensor system; - detecting (S3) glasses acceleration information by way of a glasses inertial sensor system (38) in the smartglasses (3); and the method is characterized by the further step of: - determining (S4) the installation pose of the vehicle inertial sensor system (38) on the basis of the absolute glasses poses (P), the acceleration information of the smartglasses (3) and the acceleration information of the vehicle.

2. Method according to Claim 1, wherein the installation pose of the vehicle inertial sensor system (38) is used to provide for calibration of the installation pose of the vehicle inertial sensor system (38) and / or to improve tracking of the smartglasses (3) or location of the vehicle.

3. Method according to Claim 1 or 2, wherein the acceleration information of the smartglasses (3) and / or the acceleration information of the vehicle each comprise one to three translational accelerations and / or one to three rotational accelerations with respect to at least one spatial direction.

4. Method according to one of Claims 1 to 3, wherein the installation pose of the vehicle inertial sensor system (38) is identified on the basis of minimization of an error function of an acceleration balance from the change of pose of the absolute glasses poses (P), the acceleration information recorded by the glasses inertial sensor system and the acceleration information recorded by the vehicle inertial sensor system.

5. Method according to Claim 4, wherein the error function takes account of a position error (errorposition) and an orientation error (errororientation) of the vehicle inertial sensor system (38), the position error of the vehicle inertial sensor system (38) being identified on the basis of a tangential acceleration (termacc,coriolis), a Coriolis acceleration (termacc,coriolis) and a centripetal acceleration (termacc,centripetal).

6. Display system for determining an installation pose of a vehicle inertial sensor system (38) in a vehicle using smartglasses (3) that are movable in the vehicle, the display system being designed to: - determine a glasses pose of the smartglasses (3) using a vehicle-fixed tracking system in order to obtain an absolute glasses pose (P) in a vehicle coordinate system at various times; - detect acceleration information of the vehicle externally to the glasses by way of the vehicle inertial sensor system; - detect glasses acceleration information by way of a glasses inertial sensor system (38) in the smartglasses (3); - determine the installation pose of the vehicle inertial sensor system (38) on the basis of the absolute glasses poses (P), the acceleration information of the smartglasses (3) and the acceleration information of the vehicle.