Method for providing information about road users

Abstract

The present invention relates to a method for providing information about road users (T1, T2) around a vehicle detected using on-board sensors. According to the present invention, a data structure (D) is provided that represents the detected information as vectors. Each vector is assigned to a cell (B1-Bn, BB1-BBm) of a predetermined vehicle fixed occupancy grid (B, BB) in which the road user (T1, T2) described by the respective information is located. Each vector includes at least the coordinates of the assigned cell (B1-Bn, BB1-BBm) in which the respective road user (T1, T2) is located, a speed vector representing the speed (v) of the respective road user (T1, T2), a timestamp representing the time (t1, t2) of detection of the respective road user (T1, T2), and an object class representing the type of the respective road user (T1, T2). The present invention further relates to a method for driving an autonomously capable vehicle (F1, F2).

Description

[Technical Field]

[0001] The present invention relates to a method for providing information about road users around a vehicle detected using on-board sensors.

[0002] The invention further relates to a method for operating an automated vehicle, in particular a highly automated vehicle or a vehicle capable of autonomous driving. [Background technology]

[0003] From the following patent document 1, a method for providing an occupancy map for a vehicle is known, in which a driving situation of the vehicle is calculated by a calculation device from data around the vehicle detected by a plurality of sensor devices, and the configuration of the occupancy map is adapted to the driving situation, The occupancy map has a plurality of cells arranged in a grid, which are adapted to the driving situation depending on the driving situation of the vehicle.

[0004] Furthermore, Patent Document 2 discloses a method for describing the surroundings of a vehicle, in which surrounding data is detected and stored in a hierarchical data structure, and objects are identified in the surroundings. The relevance of the objects to their applications is calculated, and the hierarchical data structure is refined in areas where objects are detected with their application relevance. In this case, the surrounding data is registered as sensor measurement data in an occupancy grid to obtain a probabilistic surroundings representation. Each cell of the occupancy grid contains an occupancy probability calculated for that location based on the sensor measurement data. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] German Patent No. 10 2013 210 263 A1 Specification [Patent Document 2] German Patent No. 10 2010 011 629 A1 Specification Summary of the Invention [Problem to be solved by the invention]

[0006] The object of the present invention is to provide a novel type of method for providing information about road users around a vehicle that are detected using on-board sensors. [Means for solving the problem]

[0007] This problem is solved according to the invention by a method for providing information about road users around the vehicle detected using on-board sensors, having the features of claim 1, and a method for driving an autonomously capable vehicle, having the features of claim 7.

[0008] Advantageous embodiments of the invention are the subject matter of the dependent claims.

[0009] In a method for providing information about road users around a vehicle detected using on-board sensors, the detected information is provided as a data structure representing a vector describing each road user. Each vector is assigned to a cell of a predetermined vehicle fixed occupancy grid in which the road user to which the information relates is located. Each vector contains at least the coordinates of the assigned cell in which the road user is located, a speed vector representing the speed of the road user, a timestamp representing the time of detection of the road user, and an object class representing the type of the road user. The information about the road users provided as vectors is preferably organized into data fields and transmitted to a backend server.

[0010] The method allows tracking of road users throughout the so-called "Operational Design Domain" (ODD). When the information present on the back-end server is provided to the fleet, a uniform, highly detailed picture of the traffic situation in terms of temporal and spatial resolution can be generated for the entire fleet. This allows the method to enable safer route planning, earlier collision avoidance and therefore more uniform driving of automated vehicles.

[0011] In a possible embodiment of the method, additional information is transmitted to the backend server, such as vehicle position and / or vehicle orientation and / or occupancy grid information regarding the definition of the occupancy grid, so that the backend server can perform a reliable and accurate conversion of the information obtained by the vehicle into a global coordinate system and provide it to other vehicles.

[0012] For this purpose, in a possible further embodiment of the method, the backend server performs a coordinate transformation based on the additional information, in which the vectors received by the vehicle as data fields are transformed from the vehicle fixed occupancy grid of this vehicle into a predetermined global location fixed occupancy grid.

[0013] In a possible further embodiment of the method, the vectors converted to a global occupancy grid are provided to other vehicles for calling, so that the other vehicles can use the information for their own specific driving modes.

[0014] In a further possible embodiment of the method, during the transmission performed during the acquisition, the transformed vectors are transmitted as a data field from the back-end server to at least one further vehicle, each vector being assigned to a cell of the global occupancy grid in which the road user described by the respective information is located. Each vector then includes at least the coordinates of the assigned cell in which the respective road user is located, a speed vector representing the respective road user's speed, a timestamp representing the time of detection of the respective road user, and an object class representing the respective road user's type. This supports the tracking of road users through the complete operational design space in order to generate a uniform, highly detailed picture of the traffic situation in terms of temporal and spatial resolution for other vehicles in the fleet. This allows for safer route planning relative to other vehicles, earlier collision avoidance, and therefore smoother automated vehicle travel.

[0015] In a further possible embodiment of the method, the other vehicles are driven automatically in a fleet of vehicles formed for automated driving, in particular for highly automated or autonomous driving.

[0016] A method according to the present invention for driving an automated vehicle, particularly a highly automated or autonomous vehicle, takes into account information retrieved from a backend server during automated driving of the vehicle. For automated driving of a vehicle, particularly for highly automated or autonomous driving, it is essential to detect the vehicle's surroundings. For this purpose, a sufficient detection range of the onboard sensors for detecting the vehicle's surroundings and a sufficiently wide field of view for reliable detection of the surroundings are required. In particular, in urban environments, the sensor detection range and the resulting vehicle field of view are limited. This method solves the problem of sensor field of view limitations, particularly for automated vehicles in a fleet, by making available to the automated vehicles in a fleet information about the vehicle's surroundings transmitted from at least one vehicle to a backend server, thereby providing information outside the onboard sensor detection range of the automated vehicle. This allows the vehicle to avoid being limited by the detection range of its own sensors. In this case, the vehicle can create an environment model for its trajectory planning that exceeds the detection range of its sensors based on the information retrieved from the backend server, and include areas blocked or outside the sensor detection range in the trajectory planning. Thus, long-term optimal route planning can be performed. Furthermore, the back-end server can be used as an external source to check the on-board sensors, for example, for detecting sensor anomalies. Together with the information obtained from the back-end server, the vehicle has additional information that can be used to check the functionality of the on-board sensors and identify malfunctioning sensors.

[0017] Embodiments of the invention are described in detail below with reference to the drawings. [Brief explanation of the drawings]

[0018] [Figure 1] 2 shows a schematic perspective view of an occupancy grid of vehicles and further road users at a first point in time and a perspective view of an occupancy grid of vehicles and further road users at a second point in time; [Figure 2] FIG. 1 is a schematic diagram of an occupancy grid for a city area. [Figure 3] FIG. 1 is a schematic diagram of a coordinate transformation of a vehicle fixed occupancy grid and its contents onto a global location fixed occupancy grid. [Figure 4] FIG. 10 is a schematic diagram of a global location fixed occupancy grid transformed from two vehicle fixed occupancy grids for two vehicles. [Figure 5] FIG. 2 is a schematic diagram illustrating the provision of information detected using on-board sensors about road users around the vehicle to a back-end server and the retrieval of this information from the back-end server.

[0019] In all the figures, corresponding parts are given the same reference numerals. DETAILED DESCRIPTION OF THE INVENTION

[0020] Figure 1 shows a perspective view of a vehicle F1 at a first point in time t1, an occupancy grid B, also called an occupancy grid, at the center of the vehicle F1, and a further road user T1, as well as a perspective view of the occupancy grid B of the vehicle F1 and a further road user T1 at a second point in time t2 following the first point in time t1.

[0021] In this case, road user T1 moves at a speed v.

[0022] The vehicle F1, for example, belongs to a fleet of vehicles and is configured for automated driving, in particular for highly automated or autonomous driving. In such automated driving, predicting the behavior of road users T1 is a major challenge. In complex traffic scenarios, it is often difficult to track and predict the behavior of all recognized road users T1. This also requires a large computational load.

[0023] To predict the behavior, an occupancy grid B is used herein, in which the vehicle F1 has a vehicle-centered coordinate system consisting of coordinates x, y, and z. The x coordinate in this case always points forward. The other axes, designated y and z coordinates, are perpendicular to the x coordinate. A grid or lattice is created around the vehicle F1, statically formed all around the vehicle F1, moving with the vehicle F1 and within the detection range of on-board sensors not shown in detail.

[0024] The vehicle F1 uses its sensors to determine the positions of all road users T1, e.g., pedestrians, cyclists, cars, freight vehicles, buses, etc., and their relative distances to the vehicle F1 and its surroundings. In addition to road users, it also detects and locates static objects, e.g., construction sites or other obstacles on the roadway. The detection can in this case be performed using multiple sensors and / or a combination of different sensors, e.g., radar, lidar, camera and / or ultrasonic sensors, and corresponding data processing, e.g., deep learning algorithms.

[0025] The detected road users T1 are described by a data structure D representing vectors. The vectors representing each road user T1 are formed from information detected using on-board sensors. Each vector includes, as information, data describing the cell B1-Bn of the occupancy grid B occupied by the road user T1 in the form of x, y, and z coordinates x, y, z of the relative position of the respective road user T1 with respect to the vehicle F1, where the z coordinate z describes the height of the respective road user T1. Furthermore, the vector includes a speed vector representing the speed v of each road user T1, timestamps representing the times t1, t2 of the detection of each road user T1, and an object class representing the type of each road user T1. Furthermore, the vector may additionally include a detection confidence, e.g., derived from which sensor detected the road user T1, and further information, e.g., the target of the road user T1.

[0026] The occupancy grid B is a 2.5-dimensional grid, i.e. it is a two-dimensional grid, where for every road user T1, a height is stored as a z-coordinate z in the respective vector. In this case, the two-dimensional grid can be described by a matrix, where each cell B1 to Bn is assigned a unique x-coordinate x and a unique y-coordinate y. To form the occupancy grid B, the two-dimensional grid is overlaid and / or combined with all the necessary information detected using the on-board sensors. That is, the data structure D describing the individual road users is compiled into one data field and sent to the backend server 1, as shown in Figure 5.

[0027] The cells B1 to Bn of the occupancy grid B may in this case be occupied or unoccupied. At time t1, for example, road user T1, formed as a pedestrian, is shown in cell B1 of the occupancy grid B, together with a data structure D describing road user T1. All other cells B2 to Bn are not occupied by road user T1, and data structure D is set to the value zero, which represents blank cells B2 to Bn.

[0028] The occupancy grid B in combination with the data structure D for each road user creates a representation of the surrounding environment at time t1.

[0029] According to this representation, road user T1 moves between cell B1 and adjacent cell B2 between two time points t1 and t2. A vector describing the position of road user T1 relative to vehicle F1 in the form of at least x, y and z coordinates, x, y, z, is thus changed and stored correspondingly, the relative position of traffic participant road user T1 with respect to vehicle F1, contained in data structure D.

[0030] For example, the complete occupancy grid B of vehicle F1 can be calculated by multiplying a matrix with a global vector containing all vectors consisting of the x-, y- and z-coordinates of road users T1, their respective relative positions to vehicle F1 in the form x, y, z. All unoccupied cells B1 to Bn are set to the value zero.

[0031] Information about the behavior of road user T1 is recorded during test and / or training drives of the autonomous vehicle F1. The information is then stored in a constant data flow of matrices. In this case, in particular, so-called deep learning models are used to process the information for predicting the future behavior of road user T1. For example, an artificial neural network is created with the deep learning model that processes information from the test and / or training drives. With an increasing amount of information, the model for predicting the future behavior of road user T1 for multiple time points t1, t2 becomes more accurate.

[0032] The above concept is additionally extended to a city-wide global occupancy grid gB consisting of a unified global coordinate system. Figure 2 shows such a global occupancy grid gB for a city area.

[0033] In Figure 3, the coordinate transformation of the vehicle-fixed occupancy grid B and its contents onto the location-fixed global occupancy grid gB is shown as the "Operational Design Domain (ODD)." In this transformation, the entire vehicle-fixed occupancy grid B for vehicle F1 is covered with a consistent grid that is identical for both vehicles F1 and F2, which are autonomously driven within the fleet. In Figure 4, vehicle F2 is shown in more detail.

[0034] The vehicle-centered coordinate system of autonomous vehicle F1, with coordinates x, y, and z, and its associated occupancy grid B are transformed into a global coordinate system with coordinates x', y', and z'. The goal is to create a unified global occupancy grid gB consisting of all information detected by the sensors of all autonomous vehicles F1 and F2 operating within the global occupancy grid gB. Each autonomous vehicle F1 and F2 shares the information detected by its own sensors with the backend server 1 shown in Figure 5. The shared information includes a data field consisting of a data structure D describing the vehicles F1 and F2. In this case, only the occupied cells gB1-gBm of the global occupancy grid gB contain information. Unoccupied cells gB1-gBm contain a value of zero. This minimizes data flow and allows for scaling to a large fleet of autonomous vehicles F1 and F2 for the operational design domain. In this case, latency reduction, a key factor in ensuring operational capability, can be achieved.

[0035] The coordinate transformation ensures that in this case all information is available: only coordinate-dependent information, such as cell coordinates and velocity vectors, is transformed into the respective data structure D. The detected road users T1 are represented both in a vehicle-centered occupancy grid B and in a global occupancy grid gB.

[0036] As can be seen from Fig. 4, which shows the global occupancy grid gB consisting of two vehicle-fixed occupancy grids B, BB of two vehicles F1, F2 converted into this, vehicles F1, F2 are each assigned to one individual vehicle-centered occupancy grid B, BB. In this case, vehicles F1, F2 can detect the same road users T1, T2 or different road users in different vehicle-centered cells B1-Bn, BB1-BBo, respectively.

[0037] By uploading their respective occupancy grids B, BB to the backend server 1 and creating a global occupancy grid gB, the doubly recognized road users T1, T2 occupy the same cells gB1 to gBm in the global occupancy grid gB through coordinate transformation. In this case, the doubly detected road users T1, T2 obtain a higher reliability or higher detection confidence because they are detected individually by the two autonomous vehicles F1, F2.

[0038] FIG. 5 shows the provision of information detected by on-board sensors about road users T1, T2 around the vehicle to the backend server 1 and the retrieval of this information from the backend server 1.

[0039] After uploading all vehicle-centered occupancy grids B, BB to the backend server 1 at time t1, a global occupancy grid gB is calculated based on the coordinate transformation including all road users T1, T2.

[0040] Next, at an artificial timestamp t1*, the global occupancy grid gB is downloaded to all vehicles F1, F2. Each vehicle F1, F2 now receives information about all dynamic objects in the global occupancy grid gB. Static objects, such as roadway obstacles or construction sites, can be taken into account as well.

[0041] Using such additional information, each individual vehicle F1, F2 can check the information detected by itself and, for example, plan in advance that the trajectories of road users T1, T2 will intersect, as well as extend its environment model over the sensor detection range.

[0042] This allows the global occupancy grid gB to scale within the operational design domain only for a sufficient number of vehicles F1, F2. The global occupancy grid gB only contains information detected by vehicles F1, F2. Cells ngB1-gBm that do not have a vehicle-centric update are marked to be described as undetected cells gB1-gBm at a particular time t1, t2.

[0043] The information from the sensors of each vehicle F1, F2, in this case including all dynamic objects, in particular road users T1, T2, is aggregated into a global vector that includes all vectors of the relative positions of each road user T1, T2 with respect to the corresponding vehicle F1, F2 in the form of x, y and z coordinates x, y, z.

[0044] In addition to information about dynamic objects, there is also interest in additional data for safer operation of vehicle fleets, which falls into the first category, as well as non-critical information about infrastructure and traffic, which falls into the second category.

[0045] The first category of safety-critical information includes, for example, data such as the status of traffic signs recognized by the fleet, as well as anomalous static objects such as double-parked vehicles, crashed cars, and lost cargo. Especially in modern cities, non-critical information is of interest. This second category includes, for example, information about occupied parking spaces, potholes, and all information related to the optimization of infrastructure and cities through proactive learning.

[0046] To make the last mentioned safety-critical and non-critical information also available, the vehicles F1, F2 of the fleet collect real-time information via a global information network. The information stored in the occupancy grids B, BB, gB can be enriched by requested data, for example regarding available parking spaces, collected in the back-end server 1 and provided via appropriate channels, such as a parking app.

Claims

1. 1. A method for providing information about road users (T1, T2) around a vehicle (F1, F2) detected using on-board sensors, comprising: said detected information is provided as a data structure (D) representing a vector each, by means of which each vector is assigned a cell (B1 to Bn, BB1 to BBm) of a predetermined vehicle fixed occupancy grid (B, BB) in which the road user (T1, T2) described by said detected information is located, said data structure (D) comprising at least the following data: the coordinates of the assigned cell (B1 to Bn, BB1 to BBm) in which the road user (T1, T2) is located; - a speed vector representing the speed (v) of said road users (T1, T2); - timestamps representing the times (t1, t2) of detection of said road users (T1, T2); - an object class representing the type of said road user (T1, T2), - a detection reliability based on the on-board sensors detecting the road users (T1, T2), The data structure (D) is integrated into a data field and sent to a backend server (1); As additional information, the position of said vehicles (F1, F2), and / or the orientation of said vehicles (F1, F2), and / or The definition of the occupancy grid information for the vehicle fixed occupancy grid (B, BB) is sent to the backend server (1), The method is characterized in that the backend server (1) performs a coordinate transformation based on the additional information to convert the vectors received as data fields from a vehicle (F1, F2) from the vehicle fixed occupancy grid (B, BB) of this vehicle (F1, F2) to a predetermined global location fixed occupancy grid (gB).

2. 2. The method according to claim 1, characterized in that the vectors transformed into the location-fixed occupancy grid (gB) are accessible to other vehicles (F1, F2).

3. A transmission performed when data is called by at least one further vehicle (F1, F2) to the back-end server (1), - said transformed vector is transmitted as a data field of said backend server (1) to said at least one further vehicle (F1, F2); - each vector of a cell (gB1 to gBo) of said location-fixed occupancy grid (gB) in which a road user (T1, T2) described by said detected information is located is assigned, said transformed vector comprising at least the following data: the coordinates of the assigned cell (gB1 to gBo) in which the road user (T1, T2) is located; - a speed vector representing the speed (v) of said road users (T1, T2); - timestamps representing the times (t1, t2) of detection of said road users (T1, T2); - an object class representing the type of said road user (T1, T2), 3. The method of claim 2, comprising:

4. 4. The method according to claim 2 or 3, characterized in that the other vehicles (F1, F2) are driven automatically in a fleet of vehicles formed for automated driving, in particular for highly automated or autonomous driving.

Images