Alignment calibration method of inertial measurement unit in head-mounted equipment

Abstract

The invention relates to a method for aligning and calibrating an inertial measurement unit, said inertial measurement unit being contained in a head-mounted device, said method comprising the following steps: (A) wearing the device on the head of a user; (B)-performing a static calibration stage (40) comprising determining a misalignment of the first sensitive axis of the inertial measurement unit with respect to the direction of gravity while the user holds the head stationary, looks forward and gazes horizontally.

Description

Technical Field

[0001] The present invention relates to a method for aligning and calibrating an inertial measurement unit. Background Art

[0002] Today, there is an increasing trend that head-mounted devices such as in-ear headphones, over-ear headphones, smart glasses, and AR / VR headsets are equipped with integrated inertial MEMS (microelectromechanical system) sensors, such as accelerometers and gyroscopes, to assist in determining the orientation of the head. This enables various advanced applications, such as head tracking, spatial audio, head pose detection, human-computer interaction, and pedestrian navigation.

[0003] Generally, the orientation of the head is described in the head coordinate system, while the raw sensor data is provided in the body coordinate system or the sensor coordinate system. To convert the information from the sensor coordinate system to the head coordinate system, the alignment offset (Ausrichtungsversatz) or rotation or "misalignment (Fehlausrichtung)" between the two coordinate systems must be accurately determined. See the illustration in Figure 1 the figure.

[0004] Due to different head sizes (which particularly affects over-ear headphones) or ear shapes (especially for in-ear headphones), the misalignment between the two coordinate systems varies from user to user. The user may also wear the device with a slightly different orientation or position each time. These situations further complicate the estimation of the misalignment. Calibration for such misalignment is required.

[0005] For head tracking using integrated sensors, different methods for aligning the head coordinate system and the sensor coordinate system are known in the art.

[0006] Publication US2023 / 0143987 A1 discloses the idea of using in-ear headphones for head pose detection. Two in-ear headphones collect accelerometer data in a reference coordinate system associated with the in-ear headphones. Then, the raw data is rotated into a neutral reference coordinate system with a fixed orientation relative to the earth using a rotation matrix. To calculate or update the rotation matrix, a stationary period is required during which the user does not move significantly to determine the average accelerometer value. The data in the neutral reference coordinate system can be analyzed to classify the user's head position using a head pose lookup table.

[0007] Patent US 9237393 B2 discloses a method for determining orientation and user head movement via a head-mounted device with an accelerometer. A pair of in-ear headphones integrated with accelerometers provides acceleration information for detecting the direction of gravity (determining the horizontal plane), and then signals from the left and right accelerometers are used to remove non-horizontal movements and determine the angular movement of the user's head within the approximate horizontal plane.

[0008] Patents US 8644531 BB and US 9950239 BA contribute to the subject of determining head position relative to gravity, but do not show the details of the method. Publications US2014254817AA and US2018091924 AA disclose how to determine head orientation based on a comparison between two accelerometer signals or by the positions of two in-ear headphones.

[0009] In all the methods mentioned, some of these methods obtain head orientation information from directions such as the direction of gaze or yaw movement. Other methods require integrating at least two sensors into a pair of in-ear headphones to obtain partial orientation information. In addition, there are also products on the market that support head tracking functions, such as Sony's wireless in-ear headphones WF-1000XM5. However, these products require a specific level 3 calibration procedure to obtain head orientation information. Summary of the Invention

[0010] The object of the present invention is to efficiently align the head coordinate system and the sensor coordinate system in a fast and user-friendly manner, such that head orientation is accurately tracked during use and the need for user intervention is also reduced compared to methods according to the prior art.

[0011] The present invention relates to a method for aligning and calibrating an inertial measurement unit, which is included in a head-mounted device, and the method comprises the following steps:

[0012] (A) - Wear the device on the user's head;

[0013] (B) - Perform a static calibration level, including determining the misalignment of the first sensitive axis of the inertial measurement unit relative to the direction of gravity when the user keeps the head stationary, looks straight ahead, and gazes horizontally.

[0014] An advantageous example of the present invention provides that in step (C), a dynamic calibration level is performed, including determining the misalignment of the second sensitive axis of the inertial measurement unit when the user tilts the head left and right or raises and lowers the head.

[0015] Another advantageous example of the present invention provides that in step (C), a mathematical correction of the misalignment of the second sensitive axis of the inertial measurement unit is performed using device-specific orientation information.

[0016] Particularly advantageously, in step (B), device-specific orientation information is provided for the static calibration level, and as a result of step (B), updated device-specific orientation information is provided for step (C).

[0017] To overcome the limitations of the previously mentioned existing methods, the following is proposed: This semi-automatic calibration method uses a 6-axis IMU consisting of a 3-axis accelerometer and a 3-axis gyroscope to estimate three-dimensional misalignment information. With this method, the user only needs to keep the head still and look straight ahead for a short period of time, and then the system analyzes all misalignment information and eliminates the need for further head movement during the calibration process. Since the user only needs to perform a minimal routine, this novel method is called semi-automatic misalignment calibration. The method offers several advantages, including: The orientation of the head is accurately obtained in all three directions (i.e., the gaze direction, pitch (Nicken), and roll (Rollen)). The user does not need to perform specific or complex head movements. Signal synchronization between two sensors is not required because only one sensor is needed. Even under dynamic conditions, the gyroscope provides highly accurate alignment.

[0018] The main idea behind the semi-automatic calibration invention is to enable the end user of a head-mounted device with head orientation tracking features (e.g., in the context of 3D audio, AR / VR, or human-machine interaction) to calibrate the system simply by keeping the head still for a short period of time. No further head movement is required to align other axes in the system. Instead of relying on further head movement, a rough or partial knowledge of the sensor placement inside the device is used to provide a predefined input. In an implementation example, this is a vector, which will be referred to as "VektorX" below, and then this vector is used for calculations to complete the full misalignment calibration between the sensor coordinate system and the head coordinate system.

[0019] This solution improves the user experience because it requires less interaction from the end user, which provides a more user-friendly plug-and-play or out-of-the-box experience. The present invention requires some preliminary knowledge about the approximate sensor orientation in the head-mounted configuration beforehand. This knowledge can be obtained from the device characteristics. This information is used to derive a technically accurate enough setting, namely VektorX. Description of the Drawings

[0020] Figure 1 Schematically shows the misalignment between the head coordinate system and the sensor coordinate system.

[0021] Figure 2 Shows a flowchart of a two-stage misalignment calibration method, which includes a static part and a head pitch movement.

[0022] Figure 3Shows a head and an earcup headphone and their respective coordinate systems.

[0023] Figure 4 Shows a flowchart of a single-stage misalignment calibration method, which includes a static part and a mathematical correction.

[0024] Figure 5a 、 Figure 5b and Figure 5c Shows an update of the device-specific orientation "VektorX" resulting from single-stage misalignment calibration.

[0025] Figure 6 Schematically shows a method for aligning and calibrating an inertial measurement unit in a head-mounted device according to the present invention. Detailed implementation

[0026] Figure 1 Schematically shows the misalignment between the head coordinate system and the sensor coordinate system. The coordinate axes X, Y, and Z are assigned to the head. The head-mounted device (here the in-ear headphone 100) includes an inertial measurement unit (IMU) 10, which has three sensitive axes that define the coordinate system X', Y', Z'.

[0027] Generally, the orientation of the head is described in the head coordinate system, while the raw sensor data of the inertial measurement unit (IMU) is provided in the body coordinate system or the sensor coordinate system. To convert information from the sensor coordinate system to the head coordinate system, the alignment offset or "misalignment" between the two coordinate systems must be accurately determined.

[0028] Due to different head sizes (which particularly affect earcup headphones) or ear shapes (especially for in-ear headphones), the misalignment between the two coordinate systems varies from user to user. The user may also wear the device in a slightly different orientation or position each time. These situations further complicate the estimation of misalignment. Both the sensor coordinate system and the head coordinate system described here are right-handed systems.

[0029] The head coordinate system is defined as:

[0030] X axis: Points from the left ear to the right ear.

[0031] Y axis: Points from the center of the head to the nose.

[0032] Z axis: Points from the center of the head upward.

[0033] This Cartesian coordinate system forms a right-handed system with an X-Y plane.

[0034] For head tracking using a head-mounted inertial sensor, the alignment of the head coordinate system and the sensor coordinate system is a prerequisite.

[0035] Figure 2 A flowchart of a two - stage misalignment calibration method is shown, which includes a static part and head pitch movement.

[0036] The raw acceleration data 11 and the raw rotation rate data 12 are fed into the sensor calibration device 20. The sensor calibration device includes an accelerometer sensor calibrator 21 and a gyroscope sensor calibrator 22. The calibrated sensor data 30 is fed into the static calibration stage 40. In this static calibration stage, tilt correction is achieved by keeping the head still and looking straight ahead. The updated angular rate 50 is fed into the dynamic calibration stage 60. In this dynamic calibration stage, gaze direction correction is performed by tilting the head up and down. A misalignment correction quaternion 70 is provided as the output.

[0037] This two - stage calibration method includes a first static stage and a second dynamic stage of user interaction and is an improved version of the method disclosed in DE102020208283A1.

[0038] Since both the head coordinate system and the sensor coordinate system are right - handed 3D systems, at least two of the three axes must be aligned to achieve a complete alignment of the two systems. The first step involves using gravity information to align the Z - axis. When the user keeps the head still, looks straight ahead and gazes horizontally, the Z - axis is aligned with the Earth's gravity vector. In this case, the accelerometer data (gravity vector measurement) can be used to calculate the tilt of the sensor, i.e., the pitch orientation and roll orientation of the device. It is recommended that the user keep the head in this state for a few seconds so that the average value of the accelerometer data can provide a good estimate of the tilt. Then, the Z - axis is aligned by performing tilt correction, i.e., using the calculated pitch angle and roll angle to reverse the rotation. The correction is calculated as a quaternion because rotation quaternions are used to represent orientations.

[0039] There are different methods to align the second axis. One scenario is to find the angle between the X - axis or Y - axis of the two systems by performing specific head movements (such as tilting the head up and down or left and right). During a movement such as nodding, the angular rate provided by the gyroscope should be mainly around the X - axis of the head, while the angular rate around the Y - axis should be close to zero. When tilting the head left and right, the angular rate should be mainly around the Y - axis of the head, while the angular rate around the X - axis should be close to zero.

[0040] By applying the tilt correction quaternion obtained from the first step to the gyroscope raw data, the updated angular rate can be used to estimate the angle between the X - axis or Y - axis, i.e., gaze direction correction. This step is referred to as dynamic calibration in this article. Then, the complete misalignment correction quaternion can be calculated by multiplying the correction quaternions obtained from these two steps. The flowchart of this two - stage calibration is in Figure 2As shown, it also includes an optional block for the sensor calibration device 20 (which will be described in detail in the next section).

[0041] This two - stage calibration method offers multiple advantages, whereby it becomes device - independent and user - independent and is effective every time. For any type of head - mounted device with an integrated 6 - axis IMU, i.e., in - ear headphones and over - ear headphones, the user can obtain the misalignment - correction quaternion for their own situation by triggering this calibration.

[0042] The correction quaternion is specific to the user's usage situation, that is, the effects of the user's different head sizes and ear shapes have been incorporated into this correction quaternion. Additionally, regardless of whether the user is wearing the device or not, calibration can be triggered each time the user wears the device to obtain an accurate correction quaternion. All these features improve the robustness, reliability, and stability of this method in any situation.

[0043] Although the two - stage calibration method is effective, it may not be considered user - friendly. The user must follow specific instructions for appropriate head movements. This may reduce the user experience when attempting the head - tracking function, especially in the case of expecting a plug - and - play device. To address this issue, the present invention adopts a novel method called a semi - automatic calibration scheme to simplify the calibration procedure and minimize the user's workload.

[0044] Generally speaking, the semi - automatic scheme only requires the static steps of two - stage calibration. The dynamic calibration is replaced by a mathematical correction based on assumptions and hardware design information.

[0045] For many head - mounted devices (such as over - ear headphones), the user mainly adjusts the headphones rotationally around the X - axis of the head (see Figure 3 ), while the rotations around the other two axes are negligible. This means that the X - axis of the head can be considered to be in a fixed direction in the sensor coordinate system and thus can be described as a known vector, i.e., VektorX. This prior information can be used to implement a mathematical correction to align the second axis after the Z - axis alignment.

[0046] Figure 3 An example of the head and over - ear headphones and their corresponding coordinate systems is shown. Figure 3 The over - ear headphones 200 are shown, where the X - axis of the head may be in a fixed direction in the device or the sensor unit 10. As an example, the X - axis of the head (solid line) is aligned with the Z - axis of the sensor (dashed line), and the sensor is an inertial measurement unit included in the over - ear headphones.

[0047] Figure 4 A flowchart of the single - stage misalignment calibration method is shown, which includes a static part and a mathematical correction.

[0048] The raw acceleration data 11 and the raw rotation rate data 12 are fed into a sensor calibration device 20. The sensor calibration device comprises an acceleration sensor calibrator 21 and a gyroscope sensor calibrator 22. The calibrated sensor data 30 is fed into a static calibration stage 40. In addition, device-specific orientation information 41, i.e., VektorX, is fed into the static calibration stage. In this static calibration stage, tilt correction is achieved by keeping the head still and looking straight ahead. An updated specific orientation information 51 is generated, which is the updated VektorX, and is provided for mathematical correction 61. In this dynamic calibration stage, gaze direction correction is performed by looking up and down. A misalignment correction quaternion 70 is provided as output.

[0049] This single-stage calibration method consists of only a single static phase of user interaction and automatically performed mathematical corrections. Thus, this method produces a semi-automatic misalignment calibrator for estimating misaligned quaternions.

[0050] To achieve the best head tracking performance, sensor calibration setups for the accelerometer and gyroscope are also included in the figure (but they are optional). The acceleration can be corrected, for example, by a 6-position calibrator, which estimates the offset and sensitivity on all three axes. For other less demanding applications, a simple accelerometer calibrator can also meet the minimum requirements. Gyroscope offset calibration is highly recommended for head tracking, because alignment errors accumulate over time during the angular rate integration process. Gyroscope offset calibration is simple. When the device is stationary for a few seconds, the gyroscope offset can be estimated from the raw angular rate. The concept of sensor offset calibration is not directly related to the misalignment calibration algorithm, but it is recommended to use them for head tracking and similar applications.

[0051] Figure 5a , Figure 5b and Figure 5c The update of the device-specific orientation "VektorX" resulting from a single-stage misalignment calibration is shown. The head coordinate system (solid line) and the sensor coordinate system (dashed line) are shown. The device-specific orientation vector is marked by a circle.

[0052] Figure 5a It shows the initial stage where the head coordinate system and the sensor coordinate system are completely misaligned.

[0053] Figure 5b The head coordinate system and the sensor coordinate system are shown with the Z axes aligned. Figure 5c The head coordinate system and the sensor coordinate system are shown perfectly aligned.

[0054] As in the first step of the two - stage calibration, the Z - axis is first aligned between the two reference systems (Bezugssystemen). Here, the user must keep the head stationary for a few seconds and look straight ahead (keep the line of sight horizontal). The average value of the acceleration (i.e., the gravity vector) is used to calculate the inclination (pitch and roll) of the sensor. The misalignment correction quaternion q1 during this period is obtained by rotating with the pitch and roll reversed. VektorX(v) should also be updated in the reference coordinate system during this period. See Figure 5 and update it as follows:

[0055]

[0056] The X - axis of the head can be described by the unit vector v f with the value [1; 0; 0]. The angle θ between the X - axis of the head and the X - axis of the system during this period can be calculated from these two vectors:

[0057]

[0058] If v i and v f are both unit vectors, then the estimate of θ can be simplified to

[0059]

[0060] Based on the axis - angle representation, the correction quaternion can be written as

[0061]

[0062] The final correction quaternion q is calculated as

[0063] q = q2q1

[0064] Configuration of the device - specific orientation information VektorX

[0065] It is recommended to remap the X - axis of the head to the sensor system at the final configuration stage of the device to obtain the best VektorX. This is the ideal way for device manufacturers (such as headphone designers) to obtain the most accurate VektorX for their own products. For some devices, such as over - ear headphone products, which have a printed circuit board (PCB), in which the IMU sensor is integrated in the center of the ear pad, the printed circuit board (sensor board) is perpendicular to the X - axis of the head. This means that the X - axis of the head is almost aligned with the Z - axis of the sensor (since the Z - axis of the sensor comes out of the plane perpendicular to the printed circuit board). See Figure 3, so VektorX can be approximated as [0; 0; 1] or [0; 0; -1]. Based on data-driven evaluation, semi-automatic calibration can utilize this approximation to achieve an accuracy within 5 degrees, which meets the requirements of many applications (e.g., the most common spatial audio applications).

[0066] The semi-automatic calibration method was evaluated using different configurations (real products and earmuff headphones with additional experimental sensor boards) and different users. It has been demonstrated that when VektorX is configured to two decimal places, the method can achieve performance comparable to two-stage calibration. Additionally, another advantage compared to two-stage calibration is that the semi-automatic scheme shows lower variance of the corrected quaternions from different users / testers because the biases from the dynamic calibration step are filtered out. Therefore, if the user can provide a good estimate of VektorX for their headset, the semi-automatic scheme can be used as a very user-friendly solution for obtaining accurate misalignment information between the head coordinate system and the sensor coordinate system.

[0067] The semi-automatic scheme can also potentially be used for in-ear headphones. However, it is important to ensure that the in-ear headphones also meet the assumptions made for the semi-automatic scheme. From a design perspective, the sensor board of the in-ear headphones can be kept perpendicular to the X-axis of the head or parallel to the Z-axis of the head.

[0068] Figure 6 An example of a method for aligning and calibrating an inertial measurement unit in a headset according to the present invention is schematically shown. The method includes the following steps:

[0069] (A) - Wear the device on the user's head.

[0070] (B) - Perform a first static calibration stage, including determining the misalignment of the first sensitive axis of the inertial measurement unit relative to the direction of gravity when the user keeps the head stationary, looks straight ahead, and gazes horizontally.

[0071] (C) - Calibrate the second sensitive axis of the inertial measurement unit, which can be done through a second dynamic calibration stage or through a mathematical correction using device-specific orientation information.

Claims

1. A method for aligning and calibrating an inertial measurement unit, the inertial measurement unit being included in a head-mounted device, the method comprising the following steps: (A) - Wear the device on the user's head; (B) - Perform a static calibration stage (40), including determining a misalignment of a first sensitive axis of the inertial measurement unit relative to the direction of gravity while the user keeps the head stationary, looks straight ahead, and gazes horizontally.

2. The method for calibrating an inertial measurement unit according to claim 1, wherein Step (C) - Perform a dynamic calibration stage (60), including determining a misalignment of a second sensitive axis of the inertial measurement unit while the user tilts the head left and / or right or the user looks up and / or down.

3. The method for calibrating an inertial measurement unit according to claim 1, characterized in that, Step (C) - Perform a mathematical correction (61) on the misalignment of the second sensitive axis of the inertial measurement unit using device-specific orientation information (41, 51).

4. The method for calibrating an inertial measurement unit according to claim 3, wherein, In step (B), provide device-specific orientation information (41) for the static calibration stage (40), and as a result of step (B), provide updated device-specific orientation information (51) for step (C).

Images