Method for generating a representation

Abstract

The invention relates to a method for generating a representation of the surroundings of a vehicle, in which ego data of the vehicle and object data which describe the surroundings of the vehicle on the basis of objects are captured as data by devices of the vehicle, wherein the data are stored in a database (30), and at least one multilayer vector network map which presents the representation of the surroundings is generated from the data.

Description

[0001] Description

[0002] title

[0003] The invention relates to a method for generating a representation of the surroundings of a vehicle and an arrangement for carrying out the method.

[0004] State of the art

[0005] Currently, vehicles are highly dependent on high-resolution maps and high-quality, instantaneously measured sensor data as a basis for perception and understanding of the situation. To simplify the task of generating a meaningful representation of the environment, there are approaches to collecting and sharing relevant ex-post data across a vehicle fleet, with a restriction to vehicles configured for communication with other road users and structures (V2X: vehicle to everything). This requires special infrastructure inside the vehicle, meaning a CCU (central control unit) must be installed, and outside the vehicle, meaning a geo-server or roadside units must be provided, depending on the communication technology.

[0006] In known applications, only accumulated data from V2X-connected vehicles or objects detected by fixed infrastructure-based sensors is used and considered to improve the local perception and planning of connected vehicles. Furthermore, all known applications use a classic object list as a representation. Disclosure of the Invention

[0007] Against this background, a method according to claim 1 and an arrangement having the features of claim 8 are presented. Embodiments emerge from the dependent claims and from the description.

[0008] A method for generating a representation of a vehicle's environment is presented. This method involves capturing ego data of the vehicle and object data describing the vehicle's environment based on objects from the vehicle's devices. These devices are typically sensors, such as cameras, acceleration sensors, etc.

[0009] The data, i.e., the ego data and the object data, are regularly stored in a local database in the vehicle. Therefore, no data originating from external devices is used. The reliability of the data is thus guaranteed and does not need to be estimated or taken into account during evaluation.

[0010] From the data, i.e. the ego data and object data, at least one multi-layer vector network map is generated, which represents the representation of the environment.

[0011] The representation of the environment thus represents a type of description of the environment that enables an evaluation of the environment and an appropriate reaction to this environment through appropriate driving behavior.

[0012] The approach presented here enables human-like, anticipatory, automated driving behavior based on an innovative flow field representation, namely the velocity vector field. No additional V2X infrastructure is required for this. To date, there are no systematic, infrastructure-free approaches that utilize accumulated environmental sensor data, particularly with a focus on a repeatedly driven commuter track, e.g., a commute to work. Accumulating and efficiently representing this data helps simplify perception tasks, such as motion model selection for short-term prediction during tracking, or planning tasks, such as long-term object prediction.

[0013] As an extension of the proposed flow field, intentionally over-fitted AI (artificial intelligence) networks can also be used for representation. Since the approach is intended to simplify the automated driving (AD) task exclusively on the commuter route, over-fitted AI networks can be a very efficient way to store and represent data on the commuter route. The data accumulated on the commuter route can be used to support the local perception and trajectory planning modules of AD vehicles and simplify the driving task.

[0014] Accumulated data can thus be used to enable faster market entry of AD systems on customer-specific commuter routes.

[0015] It further describes a specific solution for representing and using data from captured static and dynamic scene content seen on the commute every day.

[0016] The method overcomes the limitations of infrastructure-based data aggregation, requiring no V2X hardware, and simultaneously avoids the limitation of only collecting and using ego-vehicle data, in the form of fleet data from connected vehicles. In contrast, and in line with the idea of ​​accumulating data from all system interfaces, it is proposed to collect both ego-vehicle data and object data, including positions and velocity vectors relative to a global coordinate system. While vehicle fleet approaches rely on techniques such as remote databases and V2X, the presented approach provides a locally stored database and is therefore infrastructure-independent.Additionally, a highly efficient multi-layer representation, the vector network, is presented. The proposed flexible representation also allows for consideration of all relevant local object attributes, such as object classes, existence probability, variance, ex-post false positive detection rate at specific positions along the lane. Taking the data type into account, the local network cells could be represented in a class-like manner, allowing maximum flexibility. If useful, the data can be represented in the form of distributions in network-like cells. A distribution could be helpful, e.g., in the case of probabilistic representations in a long-term forecast when a road section is used for different travel directions, e.g., at intersections or roundabouts.

[0017] In this case, a simple mean value for the stored object data, e.g. for the direction of the motion vector, could be an inadequate representation.

[0018] Based on an ex-post analysis, a meaningful and efficient representation of the environment can be generated. Expanding the dataset with object data makes it possible to address and alleviate some major challenges in inner cities, such as:

[0019] For perception:

[0020] Generating suitable motion models for object short-term prediction, as applied for object tracking and subsequent motion planning.

[0021] A robust estimation of local perception quality, variances, existence probabilities, e.g. to exclude phantom measurements.

[0022] For planning: an expectation of objects behind occlusions, a robust long-term prediction of objects in complex scenarios, including multiple behavior options or behavior types, also influenced by local infrastructure, e.g., right-of-way, traffic lights, a fallback path for driving without high-resolution maps, the vector network presented here is used as input for the mapless driving-KL network.

[0023] The presented method aims to generate a standardized and highly generic representation of real-world traffic data. An important advantage of the approach is that it does not require a V2X infrastructure and yet is capable of collecting a large amount of relevant data. Since object data and ego data are used, a

[0024] experience a multiplier effect, which means that after two to three passes through the commuter lane, the database becomes usable due to the use of the objects seen in the environment.

[0025] Advantages of the method, at least in some of the embodiments, are:

[0026] Object data is taken into account in addition to ego vehicle data, which enables significantly earlier availability of the database and therefore application in low-traffic areas.

[0027] Real-time data storage and retrieval is supported, allowing accumulation of live data with previously acquired data to achieve greater robustness.

[0028] The representation of the object traffic data is independent of a topological model, making it highly reusable across different applications and environments. The representation supports the addition of scenario or environment labels, such as traffic light status, weather conditions, high-density pedestrian areas, highways, etc., enabling easy integration into various AD applications.

[0029] In terms of safety, early detection is enabled in case of a decrease in sensor quality or non-systematic perception problems.

[0030] In addition to existing vehicle fleet approaches, the extended data sets can also be stored and shared with a common database, resulting in increased data accumulation and a more robust database, and further facilitating application in low-traffic areas.

[0031] The presented arrangement is configured to carry out the presented method and has a corresponding evaluation unit. The arrangement can be implemented in hardware and / or software and can also be integrated into a control unit of a vehicle, in particular a motor vehicle, or can be designed as such a control unit. The software can be provided by a corresponding computer program, which in turn can be stored on a computer-readable storage medium.

[0032] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0033] It is understood that the features mentioned above and those to be explained below can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the present invention.

[0034] Brief description of the drawings Figure 1 shows a diagram of the structure of a system with a mode for storing and a mode for retrieving or recovering data.

[0035] Figure 2 shows a visualization of the velocity vector field around an ego vehicle on a highway.

[0036] Figure 3 shows another visualization of the velocity vector field around an ego vehicle on a highway.

[0037] Figure 4 shows a multi-layer network map representing two-dimensional velocity vectors with their counter reading at a specific cell position.

[0038] Figure 5 shows a street scene to illustrate an embodiment of the presented method.

[0039] Figure 6 shows another street scene to illustrate an embodiment of the presented method.

[0040] Figure 7 shows another street scene to illustrate an embodiment of the presented method.

[0041] Figure 8 shows another street scene to illustrate an embodiment of the presented method.

[0042] Embodiments of the invention

[0043] The invention is illustrated schematically in the drawings using embodiments and is described in detail below with reference to the drawings.

[0044] There are two main modes required for the proposed method, which can run simultaneously, namely the storage mode and the retrieval mode, as shown in Figure 1.

[0045] Figure 1 shows a diagram of a system 10 with a storage mode 12 and a retrieval mode 14. In storage mode 12, object data 22 is input as input 20 into a database node 24 for publishing, where it is filtered and stored (block 26). The data is then stored in a database 30, particularly a local database (block 32).

[0046] In recovery mode 14, 50 services 52 are activated at startup. Data is retrieved (block 54) by a query or search (block 55) in the database 30, a response is received from the database, and a global network map of average speeds is generated (block 56).

[0047] In addition, when an ego position changes (reference number 60), the ego position 64 is entered as input 62 into a demo node 70. A service with the new ego position is called in this node (block 84). This service extracts a submap from the global network map (block 80) and returns it to the demo node 70. The submap is displayed in this node (block 82). The following is performed for modes 12 and 14:

[0048] • Storage mode 12: Live data from perception modules and fusion modules are recorded and filtered data is stored in the database 30.

[0049] • The current implementation

[0050] Uses various input signals and extracts the required content from these input signals, filters the data based on various pre-set criteria, e.g. distance from object to ego-vehicle, speed of the object, etc., transforms the data to a global localization coordinate system, saves the raw data in a local database. • Possible beneficial extensions save general scenario / environment data, such as traffic light status, weather conditions, etc. This can help generate different network maps or layers based on the requested labels, which provides a better understanding of the situation, e.g. distinguishing object behavior depending on the current traffic light status, red, yellow, green, by switching between different layers of the network map, save the raw data to a remote database or regularly update a remote database, e.g.Using Wi-Fi after the vehicle is parked: this allows a fleet of vehicles to benefit from data from others, generating more data, making data accumulation more robust against outliers and errors. Data could also be represented in AI networks, which are used for the local area.

[0051] The contents of the computer lane are overfitted. In general, overfitting is undesirable. Since the proposed approach aims to simplify an AD only on specific lanes (known commuter routes), intentionally overfitted KL networks can be a good and efficient solution for representing the data on the commuter lane.

[0052] Retrieval Mode 14: This reads the raw data from the database and accumulates the data, then this mode represents the data in a multi-layer network map, which is also called distribution of movement in cells.

[0053] • Current implementation:

[0054] To accumulate the data in a specific area, e.g. 20 * 20 cm 2 around a specific position, the data in this area is averaged.

[0055] Represented data concern the position and velocity vector of objects, see Figures 2 and 3.

[0056] A multi-layered network map is used, where each cell in the network map represents a specific global real-world position and contains the average velocity vector at that position, along with the number of objects that have passed through that position, see Figure 4.

[0057] Each cell in the multi-layer network map can hold a float value, so three layers are required to represent a three-dimensional velocity vector, with the three layers holding the x-, y-, and z-vector values, respectively.

[0058] A large, e.g., 5 km * 5 km, multi-layer network map is generated when the system starts and published with the data available in database 30.

[0059] A small, e.g., 200 m x 200 m, multi-layer network map is extracted from the large network map, centered around the ego position, and constantly updated along the path. This allows for a much smaller message size.

[0060] A service is created that is responsible for handling the generation of the larger and smaller network maps on demand. Potential beneficial extensions accumulate the data by calculating the distribution of the data over a specific area: this allows for a better understanding of the raw data and the objects in the surrounding environment, this allows for better visualization of where the data is most dense, and uses an object-oriented programming approach when representing the data in a two-dimensional network map. Each cell holds a pointer to a class object, and this class can be easily customized to hold various information, e.g., scenario labels, speed data, etc. This allows for a high degree of adaptability for different applications.

[0061] A large network map is generated around the current ego position at system startup, and a smaller one whenever the position changes. Another large network map is generated if the previously generated large network map is out of range. This can also be a similar approach for octomaps (a representation that allows the use of different spatial resolutions: fine for high data density, coarse for low data density). This prevents the generation of large network maps and therefore reduces memory requirements.

[0062] Different resolutions can be set for the network map depending on the environment and scenario: a higher resolution for pedestrians in the area of ​​a crosswalk, lower resolutions for trucks on highways, etc. This reduces the memory requirements for the generated network maps. Figure 2 shows a visualization of a velocity vector field around an ego vehicle 100 on a highway 102. The illustration also shows a number of objects 104, 106, 108, and 110 that are also visualized.

[0063] Figure 3 shows a further visualization of a velocity vector field around an ego vehicle 150 on a highway 152.

[0064] Figure 4 shows a multi-layer vector network map 200 representing two-dimensional velocity vectors with their counter value at a specific cell position. A first layer 202 represents the velocity x, a second layer 204 represents the velocity y, and a third layer 206 represents the counter value. A velocity vector and a counter value are represented by three cells on the three layers: x 202, y 204, and counter value 206 at the same position 208.

[0065] Some application cases are explained below to illustrate the advantages of the described procedure.

[0066] Figure 5 illustrates a first use case, where the probability of the existence of measured objects is increased to detect phantoms and the probability of missing detection is reduced. In particular, outliers can be detected.

[0067] Figure 5 shows an ego vehicle 250 on a road 252, which is being approached by another vehicle 254. The crosses 256 mark erroneously detected objects, so-called false positive objects or phantom objects. If the same false detections occur when driving through the same locations repeatedly, this knowledge can be used to influence object generation and, for example, to require a higher object existence probability at these locations. One application is the exclusion of phantom objects, which typically arise when the radar signal is reflected, see crosses 256 at the edge of the road. In a second application, the quality of object tracking is increased. Since the presented method provides ex-post movement data, it makes lanes available with their corresponding semantics, i.e. direction of travel, and old object trajectories including positions and speed profile.For newly detected objects, the velocity can be initialized earlier than usual based on the analysis of the ex-post velocity profiles of objects at or near the same position. Furthermore, motion models for objects can be optimized. The behavior of detected objects can be predicted based on their current position and the previous behavior of objects at the same position.

[0068] The third use case allows for anticipatory responses to objects that could be seen in the future, for example, behind visual occlusions, with a probability derived from the vector field representation. More specifically, based on historical object positions, objects behind occlusions can be expected, which increases certainty in some scenarios. Additionally, possible distances and merging points known from the past can be used to increase the robustness of planning in dense urban scenarios, avoiding getting stuck in one location.

[0069] Figure 6 shows an illustration of an ego vehicle 300 on a road 302 with a first vehicle 304 and a second vehicle 306 in front. A cross 308 indicates a distance. An arrow 310 indicates an overtaking maneuver.

[0070] A fourth use case allows the anticipatory inclusion of the speed of objects provided by perception in the planning of the ego trajectory. More specifically, ex-post motion data analysis enables a long-term prediction of the behavior of detected objects and thus an improved calculation of ego trajectories in motion planning. This can, for example, enable a more human-like predictive clearance of roundabouts. Figure 7 shows a roundabout 350 with an ego vehicle 352, to which a trajectory 354 is assigned, and another vehicle 360, to which a first trajectory 362 with a low probability and a second trajectory 364 with a higher probability is assigned.

[0071] A fifth use case concerns the optimization of the ego vehicle's behavior through roundabouts and around traffic islands. Based on previous trajectories of the ego vehicle and non-ego vehicles, the ego trajectory and speed profile can be optimized.

[0072] Figure 8 shows an ego vehicle 400 in front of a traffic island 402 and a roundabout 404. An arrow also indicates an ego trajectory 410. A sixth use case considers driving without a high-resolution map. Stored ex-post movement data can help emulate high-resolution maps in areas where only standard low-resolution navigation maps are available.

Claims

Claims 1 . A method for generating a representation of the environment of a vehicle (100, 150, 250, 300, 352, 400), in which ego data of the vehicle (100, 150, 250, 300, 352, 400) and object data describing the environment of the vehicle (100, 150, 250, 300, 352, 400) based on objects (104, 106, 108, 110) are acquired as data by devices of the vehicle (100, 150, 250, 300, 352, 400), wherein the data are stored in a database (30), and at least one multi-layer vector network map (200) is generated from the data, which represents the representation of the environment.

2. Method according to claim 1, wherein data is stored in the database (30) in a storage mode.

3. Method according to one of claims 1 to 2, wherein in a retrieval mode data is retrieved from the database (30).

4. Method according to one of claims 1 to 3, wherein the objects (104, 106, 108, 110) in the environment are described using the criteria class, position, speed and size.

5. Method according to one of claims 1 to 4, in which a first vector network map is initially generated around the position of the vehicle and, with each change in the position of the vehicle, a second vector network map is generated which represents a smaller area than the first vector network map.

6. Method according to one of claims 1 to 5, wherein the resolution of the at least one vector network map (200) is adjusted depending on the environment and / or the scenario.

7. The method according to any one of claims 1 to 6, wherein the at least one vector network map (200) comprises a layer (202) for a speed, a second layer (204) for a speed y and a third layer (206) for a counter value 8. Method according to one of claims 1 to 7, in which methods of object-oriented programming are used.

9. Arrangement for generating a representation of the surroundings of a vehicle (100, 150, 250, 300, 352, 400), which is set up to carry out a method according to one of claims 1 to 8 and has an evaluation unit to evaluate recorded ego data and object data.

10. Arrangement according to claim 9, which is associated with a database (30).

Images