Method for fusing visual odometry with inertial measurement unit data

Abstract

The invention relates to a method for fusing visual odometry with inertial measurement unit (IMU) data of an ego-vehicle (1) with the following steps: - Capturing an environment of the ego-vehicle (1) by means of at least one optical sensor (2) and generating sensor data, wherein the optical sensor (2) is a camera and the sensor data are consecutively captured camera images; - Applying semantic segmentation on the camera images; - Computing a visual odometry by means of optical flow; - Observing a shape and position of a road (F) and / or road elements (4) and deriving a tilt angle between an optical axis of the optical sensor (z) and the road (F); - Estimating a three axis rotation between two consecutive camera images by means of an essential matrix, which was computed through single value decomposition; - Fusing a delta rotation between the two consecutive camera images with accelerometers of the inertial measurement unit, wherein the fusion is conducted using a filter; - Adjusting rotation parameters of the essential matrix and transforming the detected objects (3) based on the fusion of the delta rotation.

Description

[0001] Method for fusing visual odometry with inertial measurement unit data

[0002] The invention relates to a method for fusing visual odometry with inertial measurement unit data.

[0003] State of the art

[0004] In some computer vision solutions, the problem of detecting objects in the surrounding of a car is still simplified by the assumption of a flat world, which means that there is no inclination of the road assumed.

[0005] The extrinsic parameters of the camera are calibrated once in an unloaded and stationary condition at the end of the assembly line of the car or in the car's service.

[0006] In some cars, costly suspension sensors are installed to measure the deflection of the springs. This is used to observe the vehicle body movements when the car is loaded or taking a curve or when it is accelerated or braked.

[0007] In some cars a 6 DoF (Degree of freedom) Sensor-Cluster is used, to identify the pitch- and roll rate.

[0008] Object of the invention

[0009] It is an object of the present invention to propose a method to determine pitch and roll angles between a vehicle body and the horizontal plane as well as the vehicle body and the road while improving the accuracy of the visual odometry estimation and reducing costs.

[0010] This object is solved by the subject matter of independent claim 1 . Further advantageous embodiments are subject matter of the dependent claims.

[0011] Description of the invention

[0012] Visual odometry estimates the relative motion of the car based on the optical flow. The optical flow is computed by recognizing recurring features between two consecutive frames. The features are taken randomly over the pixels of the image. It does not distinguish if the feature is attached to the road or the feature is beside the road such as trees, houses etc. Visual odometry is usually pure image based and does not fuse the acceleration sensors as provided by the inertial measurement unit (IMU).

[0013] If the world is assumed to be flat, the objects detected by the camera such as park-markers can appear further apart or closer than they actually are when they are drawn into a map. The scene will appear distorted.

[0014] The objects which are detected on the image sensor by the camera are transformed from the pixel coordinate system to the camera coordinate system using the intrinsic parameters. From the camera coordinate system, they are transformed to the world coordinate system by the extrinsic matrix. In this transformation, they are rotated and translated. The intrinsic and extrinsic parameters are calibrated at the end of the production. They usually are not adjusted while the car's body is tilted when it is accelerated / decelerated, driving a curve or when it is loaded. This can also cause the scene and the 3D reconstruction to appear distorted.

[0015] The suspension sensors are costly and error prone. The same applies for the 6 DoF IMU which would cause additional costs compared to the default 3 DoF IMU, which is installed in the cars in any case.

[0016] According to the invention a method for fusing visual odometry with inertial measurement unit (IMU) data of an ego-vehicle with the following steps is proposed:

[0017] - Capturing an environment of the ego-vehicle by means of at least one optical sensor and generating sensor data, wherein the optical sensor is a camera and the sensor data are consecutively captured camera images;

[0018] - Applying semantic segmentation on the camera images;

[0019] - Computing a visual odometry by means of optical flow;

[0020] - Observing a shape and position of a road and / or road elements and deriving a tilt angle between an optical axis of the optical sensor and the road;

[0021] - Estimating a three axis rotation between two consecutive camera images by means of an essential matrix, which was computed through single value decomposition;

[0022] - Fusing the delta rotation between the two consecutive camera images with accelerometers of the inertial measurement unit, wherein the fusion is conducted using a filter; Adjusting rotation parameters of the essential matrix and transforming the detected objects based on the fusion of the delta rotation.

[0023] The invention proposes a method to fuse visual odometry and the accelerometers of the IMU to have the pitch and roll angles between the vehicle's body and the horizontal plane. Having this enables a part which is needed to draw the objects into an internal 3D map and even provide extrinsic parameters.

[0024] Just furthermore it will aim to improve the accuracy of the visual odometry estimation. Especially the estimation of the relative rotation (rol l / pitch) rather than the angular changes which happened between two consecutive images.

[0025] In addition, it proposes a method to distinguish between the angles between the vehicle body and the horizontal plane and the angle between the vehicle body and the road. The latter is comparable to what is measured by the suspension travel sensor. Therefore, the proposed solution reduces the costs, since it avoids using the suspension travel sensors and uses a standard 3 DoF IMU (acceleration ax, ay and yaw-rate) instead.

[0026] The environment is captured continuously and consecutive camera images are captured by the camera.

[0027] When applying the semantic segmentation on the images, the road and objects which are further away close to the horizon are classified.

[0028] The shape and position of the road, especially the lane markers around and in front of the car is observed. From there, the tilt angle between the axis of the camera and the road is derived. For example: The tilted lane marker appears at higher pixels on the imager. This shall be used to estimate dynamic slope changes of the road.

[0029] The rotation in three axis (roll, pitch yaw) is estimated by the essential matrix which was computed through single value decomposition. yiTEy2 = 0 and E=R[t]x

[0030] In this case, the translation (longitudinal velocity) of the car is taken from the wheel speed sensors and visual odometry shall estimate the rotational movements only. Knowing the precise circumference of the tire, the vehicle speed can be easily estimated through the rotation of the four wheels. Fusing the delta rotation (between two images) based on the optical flow taken close to the horizon with the accelerometers ax, ay of the IMU. Optionally, the delta rotation around three axis can be fused with the rotation rates of the 6 DoF IMU, if available.

[0031] Alternatively or additionally the fusion of the delta rotation can be based on the optical flow from the road using a spring damper model. The model can be refined considering the exact pitch and roll axis.

[0032] The linear spring / damper model is defined: xe: Angle between camera and road as identified by observing the road. xa: Angle between camera and the horizontal plane. The result of the fusion between visual odometry and the accelerometers of the IMU D: Dampening coefficient coo: characteristic angular frequency

[0033] According to the model, the xais the result or output of the stimulation of xe. But this will be used to define the characteristic of the connection between the two angles for a fusion in a Kalman-Filter.

[0034] This transfer function is turned into the state space from which defines the linear Kalman-Filter:

[0035] The actual measurement (y(t)) in this Kalman-Filter which is provided in the update step is the fused output of the above fusion between the accelerometer measurements and the visual odometry computed on the optical flow from the horizon. The angle between camera and road (xe) is fed in the prediction step as input.

[0036] In a preferred embodiment the fusion is conducted by using a complementary filter or a Kalman filter. The fusion can be conducted using a complementary filter. The acceleration measured by the IMU is compensated by the acceleration caused by braking / accelerating or cornering the car. This can be estimated using wheel speed sensors. From the acceleration, the gravity vector and hence the angle to the horizontal plane can be computed. This signal can be very noisy caused by stimulation by the road. The visual odometry computes the angular changes in roll and pitch between two consecutive frames. These angular change rates will be integrated to compute roll and pitch angles. Those angles will be compared to the angles computed using the IMU accelerometers. The deviation can be subtracted from the angular change rates. Similar like a PI controller with a tuneable proportional and integrational part. If the error is high, the value which will be reduced from the angular change rates is high too. The impact of visual odometry would be reduced if it causes errors. As the visual odometry data is delayed and has a slower cycle time, the data from the IMU must be buffered. If visual odometry is updated, the buffered IMU data is taken at the same timestamp as the visual odometry. From there the complementary filter is re-sampled to the current time.

[0037] The complementary filter is the easiest example for a fusion. The data can be fused using a Kalman-Filter too. This will give better results but is less intuitive to illustrate.

[0038] In a further preferred embodiment the visual odometry is computed by means of optical flow close to the horizon.

[0039] It is further preferred that the detected transformed objects are drawn into a 3D-map. Based on the filtered angle between the camera and the horizontal plane, the rotation parameters in the extrinsic matrix can be adjusted and the detected objects transformed accordingly and drawn into the 3D map. Hence the map has three dimension the rotation can happen based on the angle to the horizontal plane. If the map was 2D, only it had to be rotated according to the angle between the camera and the road.

[0040] Further advantageous embodiments are subject matter of the drawings. Therein show:

[0041] Fig. 1 : a schematic flow chart of the method according to an embodiment of the invention; Fig. 2: a schematic view of a driving situation to apply the method according to an embodiment of the invention;

[0042] Fig. 3: a further schematic view of a driving situation to apply the method according to an embodiment of the invention.

[0043] Figure 1 shows a schematic flow chart of the method according to an embodiment of the invention. In step S1 an environment of the ego-vehicle 1 is captured by means of at least one optical sensor 2 and sensor data is generated, wherein the optical sensor 2 is a camera, and the sensor data are consecutively captured camera images. In step S2 semantic segmentation is applied on the camera images. In step S3 visual odometry is computed by means of optical flow. In step S4 a shape and position of a road F and / or road elements 4 is observed and a tilt angle a between an optical axis z of the optical sensor 2 and the road F is derived. In step S5 a three-axis rotation between two consecutive images is estimated by means of an essential matrix, which was computed through single value composition. In step S6 a delta rotation between the two consecutive camera images is fused with accelerometers of the inertial measurement unit, wherein the fusion is conducted using a filter. In step S7 rotation parameters of the essential matrix are adjusted and the detected objects 3 are transformed based on the fusion of the delta rotation.

[0044] Figure 2 shows a schematic view of a driving situation to apply the method according to an embodiment of the invention. In this figure an ego-vehicle 1 is shown driving on a road F. The ego-vehicle has at least one sensor 2, in particular an optical sensor, which captures the environment of the ego-vehicle 1. Due to acceleration or deceleration the ego-vehicle 1 can be tilted and have tilted position 1 a. With the sensor 2 attached to the vehicle, the sensor 2 has also a tilted position 2a together with the tilted position 1 a of the ego-vehicle 1 . The ego-vehicle 1 captures objects 3 with the sensor 2 in the normal position and the tilted position 1 a. The visual odometry in this case can be computed by means of optical flow. The road F and the objects 3 can be classified by applying a semantic segmentation to the images. This information can then be fused with the data from an IMU

[0045] Figure 3 shows a further schematic view of a driving situation to apply the method according to an embodiment of the invention. In the upper view the ego-vehicle 1 is driving on a non-tilted road F. The sensor 2 captures road markings 4. In the lower view the ego-vehicle 1 is driving on a tilted road F and the sensor still captures the road markings 4. Due to the tilted road F the detected road markings 4 will appear on higher pixels on the image sensor. The road F has a tilt angle a which can be determined by fusing two images of the straight and the tilted road F and the detected road markings 4.

[0046] Reference list

[0047] 1 Ego-vehicle

[0048] 1a tilted position ego-vehicle 2 Sensor

[0049] 2a tilted position sensor

[0050] 3 Object

[0051] 4 road marking

[0052] F road a tilt angle road z optical axis

Claims

Claims1 . Method for fusing visual odometry with inertial measurement unit (IMU) data of an ego-vehicle (1 ) with the following steps:- Capturing an environment of the ego-vehicle (1 ) by means of at least one optical sensor (2) and generating sensor data, wherein the optical sensor (2) is a camera and the sensor data are consecutively captured camera images;- Applying semantic segmentation on the camera images;- Computing a visual odometry by means of optical flow;- Observing a shape and position of a road (F) and / or road elements (4) and deriving a tilt angle between an optical axis of the optical sensor (z) and the road (F);- Estimating a three axis rotation between two consecutive camera images by means of an essential matrix, which was computed through single value decomposition;- Fusing a delta rotation between the two consecutive camera images with accelerometers of the inertial measurement unit, wherein the fusion is conducted using a filter;- Adjusting rotation parameters of the essential matrix and transforming the detected objects (3) based on the fusion of the delta rotation.

2. Method according to claim 1 , characterized in that, the fusion is conducted by using a complementary filter or a Kalman filter.

3. Method according to claim 1 , characterized in that, the visual odometry is computed by means of optical flow close to the horizon.

4. Method according to claim 1 , characterized in that, the detected transformed objects (3) are drawn into a 3D-map.

Images