Method and device for improved position determination of an ego vehicle using a 3D object recognition module

Abstract

Method for improved positioning of an ego vehicle (1) comprising a localization system, a 3D object recognition module and a position fusion module (41, 42, 43), comprising the following steps: a) Determining the position of the ego vehicle (1) for the current time step using the localization system, b) Determining the position of at least one neighboring vehicle (21, 22) in the vicinity of the ego vehicle (1) using the 3D object recognition module, c) Fusion of the position of the ego vehicle (1) determined in step a) with the position of at least one neighboring vehicle (21, 22) determined in step b) to a position fusion result (7) with the position fusion module (41, 42, 43), and d) Determining the position of the Ego vehicle (1) for the next time step using the localization system, taking into account the fusion result (7).

Description

State of the art

[0001] The present invention relates to a method and a device for improved position determination of an ego vehicle using a 3D object recognition module. A control unit, a computer program, and a machine-readable storage medium are also described. The invention can be used in particular for GNSS-based localization systems for autonomous or semi-autonomous driving.

[0002] Modern vehicles are typically equipped with GNSS-based localization systems for determining their position. Additionally, they may be equipped with sensors for perceiving their surroundings, such as radar, cameras, and ultrasonic sensors.

[0003] To improve positioning accuracy, some known approaches employ multiple inertial measurement units (IMUs), typically mounted on the same rigid body, allowing measurements from one sensor to be projected onto another, for example, using rigid body simulation. Other approaches utilize geometric knowledge of the environment, which can be acquired, for example, as point clouds from a lidar. These point clouds can then be used by another agent to predict its own position and pose. However, this requires high-bandwidth communication systems for information exchange, as these point clouds cannot be readily shared over conventional low-bandwidth communication systems.

[0004] Therefore, there is a desire to create a novel method for improved position determination that requires neither a rigid body simulation nor a communication system for exchanging point cloud-relevant information. Disclosure of the invention

[0005] This is achieved through a method for improved positioning of an ego vehicle, wherein the ego vehicle comprises a localization system, a 3D object recognition module and a position fusion module, and wherein the method comprises the following steps: a) Determining the position of the ego vehicle for the current time step using the localization system, b) Determining the position of at least one neighboring vehicle in the vicinity of the Ego vehicle using the 3D object recognition module, c) Merging the position of the ego vehicle determined in step a) with the position of at least one neighboring vehicle determined in step b) to produce a position fusion result using the position fusion module, and d) Determining the position of the Ego vehicle for the next time step using the localization system, taking into account the fusion result.

[0006] The described method is particularly suitable for autonomous driving. Autonomous driving can be understood as the movement of vehicles that behave largely autonomously, for example, by means of a GNSS-based localization system and / or sensors for perceiving the environment, such as radar, cameras, or ultrasonic sensors. These vehicles can be motor vehicles such as passenger cars, trucks, or other commercial vehicles, robots, or similar devices.

[0007] To carry out the described procedure, the Ego vehicle is equipped with a localization system, a 3D object recognition module and a position fusion module.

[0008] In contrast to known localization systems, where the position of an ego vehicle for the next time step is predicted recursively based on real-time measurement data and the position of the ego vehicle determined in the current time step, the localization system described here can predict the position of the ego vehicle for the next time step based on real-time measurement data and a position fusion result, whereby the position fusion result can be determined by fusing the position of the ego vehicle determined in the current time step with the position of at least one neighboring vehicle determined in the current time step using the position fusion module, and where the position of the at least one neighboring vehicle can be determined using the 3D object recognition module.

[0009] The neighboring vehicles described here are only those vehicles that are in the vicinity of the Ego vehicle and whose positions can be detected by the 3D object recognition module located in the Ego vehicle.

[0010] The localization system described here can be, in particular, a GNSS-based localization system capable of determining the position of the ego vehicle by receiving GNSS signals. GNSS is the abbreviation for Global Navigation Systems (GPS, GLONASS, Galileo, and BeiDou). The real-time measurement data consists specifically of the GNSS signals measured within the ego vehicle's field of view.

[0011] According to step a), the position of the ego vehicle for the current time step is determined using the localization system. The position of the ego vehicle can be determined from the currently measured data (i.e., real-time measurement data), such as the currently measured GNSS signals.

[0012] The 3D object recognition module described here is capable of capturing the position of at least one neighboring vehicle using an approach known from the field of computer vision. Computer vision is a field within artificial intelligence (AI) that enables computers and systems to extract meaningful information from digital images, videos, and other visual inputs—and to take action or make recommendations based on this information.

[0013] The 3D object recognition module is, for example, capable of recognizing objects in the ego vehicle's environment based on data measured by at least one sensor, typically a camera or lidar. Thus, the 3D object recognition module can not only detect an object in the ego vehicle's environment, but also determine the object's position and distinguish its type (e.g., whether it is a vehicle or a tree).

[0014] The 3D object recognition module can, in particular, be a neural network trained by deep learning, implemented on the ego vehicle, and capable of recognizing objects and assigning them to the appropriate category, as well as determining the position of the recognized objects, for example, using a 3D bounding box.

[0015] According to step b), the position of at least one neighboring vehicle in the vicinity of the Ego vehicle is determined using the 3D object recognition module. Depending on the driving scenario, one or more neighboring vehicles can be taken into account.

[0016] The position fusion module described here is designed to fuse the position of the ego vehicle determined by the localization system and the position of at least one neighboring vehicle determined by the 3D recognition module into a position fusion result.

[0017] According to step c), the position of the ego vehicle determined in step a) by the localization system is fused with the position of at least one neighboring vehicle determined in step b) by the 3D object recognition module using the position fusion module to produce a position fusion result. The position fusion result can include a fused position. This fused position can then be fed back into the localization system, allowing the position of the ego vehicle to be predicted with improved accuracy for the next time step. In particular, this fused position can be the position of the ego vehicle corrected for the determined position of at least one neighboring vehicle.

[0018] In step c), the fusion can be performed using fusion algorithms. The fusion can include providing fusion coefficients, projecting the determined position of at least one neighboring vehicle onto the ego vehicle, and determining the position fusion result.

[0019] The fusion coefficients can be the uncertainty coefficients β for the position of the ego vehicle and for the position of each neighboring vehicle and can be provided according to formulas (1), (2) and (3). β0=1σego βi=1a1ri+a2σadj,i,i=1,…,N β→=(∑i=0Nβi)−1β→

[0020] Formula (1) shows that the uncertainty coefficient for the position of the ego vehicle β0 itself depends on the positioning accuracy of the ego vehicle σ. ego depends. The uncertainty coefficient for the position of the ego vehicle β0 is indicated here by a subscript zero.

[0021] Formula (2) shows that the uncertainty coefficient for the position of a neighboring vehicle β i both from the positioning accuracy of this neighboring vehicle σ adj,i as well as the distance between the ego vehicle and this neighboring vehicle r i depends. Additionally, the positioning accuracy σ adj,i with a hyperparameter α1 and the distance r i The parameters can be weighted with a further hyperparameter α2. α1 can be set to 0.2 and α2 to 0.8.

[0022] The uncertainty coefficient for the position of a neighboring vehicle β iThe number of neighboring vehicles is denoted here by a subscript natural number i=1,...,N. For example, i=1 means that only one neighboring vehicle can be detected in the vicinity of the ego vehicle. For i=2, it means that two neighboring vehicles can be detected in the vicinity of the ego vehicle, so one can be denoted by a subscript 1 and the other by a subscript 2.

[0023] All provided uncertainty coefficients β are normalized to a unit vector according to formula (3).

[0024] The projection of the position of at least one neighboring vehicle onto the ego vehicle can be carried out according to formulas (4) and (5). ψi=atan2(xadj,i,xego) xegoadj,i=[xadj,ix+ricos(ψi)xadj,iy+risin(ψi)]

[0025] According to formula (4), the angle ψ i between the Ego vehicle x ego and a neighboring vehicle x adj,ito be determined in the line of sight. With the angle ψ i can the position of this neighboring vehicle projected along the line of sight to the ego vehicle xegoadj,i The position of this neighboring vehicle can be determined according to formula (5). The projection allows the position of this neighboring vehicle to be made available to the ego vehicle.

[0026] In the case of multiple neighboring vehicles, an uncertainty coefficient β can be assigned to each of the neighboring vehicles. i according to formulas (1), (2) and (3) and a projected position xegoadj,i according to formulas (4) and (5) such that this determined projected position is weighted with this determined uncertainty coefficient and all such weighted projected positions are superimposed to a single projected position according to formula (6). xegoadj=∑i=1Nβi⋅xegoadj,i

[0027] The determination of the position fusion result can be carried out according to formula (7). x˜ego=β0xegoGNSS+xegoadj

[0028] Formula (7) shows that the position of the Ego vehicle determined from the real-time measurement data (especially the measured GNSS signals) is weighted with the uncertainty coefficient β0 determined according to formula (1) and then with the single projected position determined according to formula (6). xegoadj The position fusion result is superimposed to obtain a position fusion result. This result can be entered into the localization system, allowing the position of the ego vehicle to be determined with improved positioning accuracy for the next time step, taking this position fusion result into account. The position fusion result according to formula (7) is the position of the ego vehicle corrected for the position of at least one neighboring vehicle.

[0029] According to step d), the position of the ego vehicle for the next time step is determined using the localization system, taking into account the position fusion result. The position fusion result can be determined, in particular, according to formulas (1) to (7).

[0030] In step d), unlike in known approaches, the position of the ego vehicle determined by a localization system based on real-time measurement data in the current time step is not fed back into the localization system. Instead, a position fusion result is used, e.g., in the form of the ego vehicle's position corrected for the position of at least one neighboring vehicle. This allows the localization system to determine the ego vehicle's position for the next time step based on the position fusion result and the real-time measurement data, such as the currently measured GNSS signals. The position of the at least one neighboring vehicle can be determined using the 3D object recognition module according to step b), and the position fusion result can be determined using the position fusion module according to step c).

[0031] The position of the ego vehicle determined in this way exhibits higher positioning accuracy. A rigid body simulation and a broadband communication system, as mentioned earlier, are not required. In particular, the method described here can be added as redundancy to the conventional method of position determination using a standard GNSS-based localization system. No complex hardware upgrades are necessary; this primarily concerns software updates.

[0032] Since the localization system determines the position of the ego vehicle step by step from real-time measurement data, steps a), b), c), and d) can be executed repeatedly if at least one neighboring vehicle can be detected in its vicinity while the ego vehicle is traveling. Although steps a) to d) are given here in a specific order, this order does not always have to be followed. For example, the individual steps can be repeated independently any number of times and / or partially omitted during repetition. At least partial temporal overlap of the steps is possible.

[0033] It is preferred if, in step a), the position of the ego vehicle is determined using the localization system from measured GNSS signals.

[0034] It is also preferred if, in step b), the position of the at least one neighboring vehicle is predicted using the 3D object recognition module and the 3D bounding box prediction approach.

[0035] It is particularly preferable if the following sub-steps are carried out in step c): i) Providing fusion coefficients, ii) Projecting the position of at least one neighboring vehicle onto the ego vehicle, and iii) Determining the positional merger result.

[0036] Furthermore, it is preferred if, in sub-step ii), the position of at least one neighboring vehicle along the line of sight to the ego vehicle is projected onto the ego vehicle.

[0037] Furthermore, a device for improved positioning of an ego-vehicle is proposed. The device comprises a localization system, a 3D object recognition module, and a position fusion module, wherein the localization system, the 3D object recognition module, and the position fusion module are mountable in an ego-vehicle. The localization system is configured to determine the position of an ego-vehicle. The 3D object recognition module is configured to determine the position of at least one neighboring vehicle in the vicinity of the ego-vehicle. The position fusion module is configured to determine a position fusion result by fusing the position of the ego-vehicle with the position of the at least one neighboring vehicle. The localization system is capable of determining the position of the ego-vehicle taking the position fusion result into account.

[0038] It is preferred if the localization system is a GNSS-based localization system with a Kalman filter, where the Kalman filter is able to determine the position of the ego vehicle based on the position fusion result and the measured GNSS signals.

[0039] It is also preferable if the 3D object recognition module is able to predict the position of at least one neighboring vehicle using the 3D bounding box prediction approach.

[0040] Furthermore, it is preferred if the 3D object recognition module is a neural network trained using deep learning.

[0041] It is particularly preferred if the position fusion module comprises a first sub-module for providing fusion coefficients, a second sub-module for projecting the position of the at least one neighboring vehicle onto the ego vehicle, and a third sub-module for determining a position fusion result.

[0042] It is particularly preferred if the first submodule, the second submodule, the third submodule and the Kalman filter are connected in series.

[0043] Furthermore, a control unit is proposed that is set up to carry out the described procedure.

[0044] Furthermore, a computer program is proposed that is used to carry out a procedure described herein. In other words, this specifically concerns a computer program (product) comprising instructions that, when executed by a computer, cause it to perform a procedure described herein.

[0045] Furthermore, it is proposed that a machine-readable storage medium be used on which the proposed computer program is stored. This machine-readable storage medium is typically a computer-readable data carrier.

[0046] The solution presented here and its technical context are explained in more detail below with reference to the figures. It should be noted that the invention is not intended to be limited by the illustrated embodiments. In particular, unless explicitly stated otherwise, it is also possible to extract partial aspects of the situations explained in the figures and combine them with other components and / or findings from other figures and / or the present description. The figures show schematically and by way of example: Fig. 1 a described procedure for a typical driving scenario and Fig. 2 a block diagram of a described procedure.

[0047] Fig. Figure 1 shows a typical driving scenario: An ego vehicle 1 is driving in one lane of a two-lane road, while a neighboring vehicle 21 drives into the same lane in front of the ego vehicle 1 and another vehicle 22 drives into the other lane in front of the ego vehicle 1.

[0048] The position of Ego-Vehicle 1 can be determined according to step a) using a localization system installed in Ego-Vehicle 1. The localization system is typically a GNSS-based localization system.

[0049] The positions of neighboring vehicles 21 and 22 can be determined according to step b) using a 3D object recognition module installed in the ego vehicle 1. Therefore, no communication systems are required for information exchange between the ego vehicle 1 and the neighboring vehicles 21 and 22.

[0050] To provide the ego vehicle 1 with the positions of neighboring vehicles 21 and 22, these positions are projected onto the ego vehicle 1 along line of sight 3 according to formulas (4) and (5). Additionally, an uncertainty coefficient for the position of the ego vehicle 1 can be determined according to formula (1), and an uncertainty coefficient for neighboring vehicle 21 and another for neighboring vehicle 22 can be determined according to formulas (2) and (3).

[0051] Thus, according to step c), the position of ego vehicle 1 determined in step a) and the positions of the two neighboring vehicles 21 and 22 determined in step b) can be fused into a position fusion result using a position fusion module installed in the ego vehicle according to formula (7). In this driving scenario, the position fusion result determined according to formula (7) is the position of ego vehicle 1 corrected for the positions of the neighboring vehicles 21 and 22.

[0052] According to step d), the position fusion result can again be entered into the localization system, so that the localization system can determine the position of the Ego vehicle 1 for the next time step based on this position fusion result and the real-time measurement data.

[0053] The position of the ego vehicle determined in this way exhibits higher positioning accuracy. A rigid body simulation and a broadband communication system, as mentioned earlier, are not required. In particular, the method described here can be added as redundancy to the conventional method of position determination using a standard GNSS-based localization system. No complex hardware upgrades are necessary; this primarily concerns software updates.

[0054] Fig. Figure 2 schematically shows an exemplary block diagram of a described procedure.

[0055] In Fig.The components 2 are a first submodule 41, a second submodule 42, a third submodule 43, and a Kalman filter 8 connected in series. The Kalman filter 8 is specifically an extended Kalman filter and can be configured in a localization system. The first submodule 41, the second submodule, and the third submodule 43 form a position fusion module that precedes the Kalman filter 8, allowing the output of the third submodule 43 to be directly fed to the Kalman filter 8.

[0056] The first submodule 41 is designed to provide the fusion coefficients 5. The fusion coefficients 5 are, in particular, the uncertainty coefficients according to formulas (1), (2), and (3). The first submodule 41 can be executed as software into which the algorithms according to formulas (1), (2), and (3) are loaded. Additionally, the distance r can be stored in the first submodule 41. i9. The fusion coefficients between the ego vehicle and at least one neighboring vehicle must be entered, which can be determined, for example, using a 3D object recognition module. The provided fusion coefficients 5 can be entered as input values ​​into the second submodule 42.

[0057] The second submodule 42 is configured to provide the ego vehicle with the position of at least one neighboring vehicle. The position of the at least one neighboring vehicle is projected onto the ego vehicle along its line of sight, resulting in a projected position 6 that can be inputted as an input value into the third submodule 43. The second submodule 42 can also be implemented as software, into which the algorithms according to formulas (4), (5), and (6) are loaded. Additionally, state information 10 can be input into submodule 42. State information 10 includes, in particular, the position of the ego vehicle determined in step a) and the position of the at least one neighboring vehicle determined in step b).

[0058] The third submodule 43 is configured to provide the position fusion result 7 in the form of a position of the ego vehicle corrected for the position of at least one neighboring vehicle, taking into account the fusion coefficients 5 provided in submodule 41 and the projected position 6 provided in submodule 42. The third submodule 43 can also be executed as software into which the algorithms according to formula (7) are loaded.

[0059] The position can be entered into the third submodule 43. xegoGNSS 11, which was determined from the real-time measurement data, such as measured GNSS signals.

[0060] Based on the position fusion result 7 in conjunction with the real-time measurement data, the Kalman filter 8 can thus predict the position of the ego vehicle for the next time step with increased accuracy and gives an optimal ego vehicle position x̂ ego12 and a covariance P̂ ego 13 out.

Claims

[1] Method for improved positioning of an ego vehicle (1) comprising a localization system, a 3D object recognition module and a position fusion module (41, 42, 43), comprising the following steps: a) Determining the position of the ego vehicle (1) for the current time step using the localization system, b) Determining the position of at least one neighboring vehicle (21, 22) in the vicinity of the ego vehicle (1) using the 3D object recognition module, c) Fusion of the position of the ego vehicle (1) determined in step a) with the position of at least one neighboring vehicle (21, 22) determined in step b) to a position fusion result (7) with the position fusion module (41, 42, 43), and d) Determining the position of the Ego vehicle (1) for the next time step using the localization system, taking into account the fusion result (7). [2] Method according to claim 1, wherein in step a) the position of the Ego vehicle (1) is determined using the localization system from measured GNSS signals. [3] Method according to claim 1 or 2, wherein in step b) the position of the at least one neighboring vehicle (21, 22) is predicted using the 3D object recognition module using the 3D bounding box prediction approach. [4] Method according to any of the preceding claims, wherein in step c) the following sub-steps are carried out: i) Providing fusion coefficients, ii) Projecting the position of at least one neighboring vehicle (21, 22) onto the ego vehicle (1), and iii) Determining the positional merger result (7). [5] Method according to claim 4, wherein in partial step ii) the position of the at least one neighboring vehicle (21, 22) along the line of sight to the ego vehicle (1) is projected onto the ego vehicle (1). [6] Device for improved positioning of an ego vehicle (1), comprising a localization system, a 3D object recognition module and a position fusion module (41, 42, 43), wherein the localization system, the 3D object recognition module and the position fusion module (41, 42, 43) are attachable in an ego vehicle (1), and wherein - the localization system for determining the position of an ego vehicle (1), - the 3D object recognition module for determining the position of at least one neighboring vehicle (21, 22) in the vicinity of the ego vehicle (1), and - the position fusion module (41, 42, 43) is set up to determine a position fusion result (7) by merging the position of the ego vehicle (1) with the position of at least one neighboring vehicle, and wherein the localization system is able to determine the position of the ego vehicle (1) taking into account the position fusion result (7). [7] Device according to claim 6, wherein the localization system is a GNSS-based localization system with a Kalman filter (8), wherein the Kalman filter (8) is able to determine the position of the ego vehicle based on the position fusion result (7) and the measured GNSS signals. [8] Device according to claim 6 or 7, wherein the 3D object recognition module is able to predict the position of the at least one neighboring vehicle (21, 22) using the 3D bounding box prediction approach. [9] Device according to any one of claims 6 to 8, wherein the 3D object recognition module comprises a neural network trained by deep learning. [10] Device according to one of claims 6 to 9, wherein the position fusion module comprises a first sub-module (41) for providing fusion coefficients, a second sub-module (42) for projecting the position of the at least one neighboring vehicle (21, 22) onto the ego vehicle (1), and a third sub-module (43) for determining a position fusion result (7). [11] Device according to claim 10, wherein the first sub-module (41), the second sub-module (42), the third sub-module (43) and the Kalman filter (8) are connected in series. [12] Control unit which is configured to carry out a method according to any of the preceding claims. [13] Computer program for carrying out a method according to any one of the preceding claims 1 to 5. [14] Machine-readable storage medium on which the computer program according to claim 13 is stored

Images