Position prediction for dynamic objects

Abstract

The invention relates to a control system and a method for predicting the position of a dynamic object. The control system (10) has a plurality of sensors (30) and a computing system (40), which is configured to combine objects (20) detected by the plurality of sensors (30) into an object list (22) by means of a first program, each element of the object list (22) including a position, a speed and a free path for each of the objects (20), and the object list (22) including a time stamp; based on a predefined number of object lists (22), another object list (23) is determined for at least part of the dynamic objects (20) by means of a second program, wherein the other object list (23) includes a time stamp for a future time and at least includes the position of the dynamic object (20).

Description

Technical Field

[0001] The invention relates to a control system for a vehicle and also to a method for predicting the position of a dynamic object, such as a pedestrian, which can be detected by sensors of the vehicle. Background Art

[0002] Detecting the dynamic surroundings of a vehicle is a task that can currently only be accomplished with very high computing power in many cases. For example, for vehicles that provide a high degree of driver support or at least partially automated vehicles, predicting the position of dynamic objects is important, in particular in order to avoid and / or prevent collisions. Summary of the invention

[0003] One aspect of the invention relates to a control system for a vehicle. The vehicle may be a land vehicle, in particular a car, a truck, a bus or a special vehicle. The control system for predicting the position of a dynamic object has a plurality of sensors and a computing system. The dynamic object may be, for example, a pedestrian, a cyclist or other traffic participant, in particular a traffic participant whose movement may strongly deviate from a straight line movement or a uniform movement. The sensors may, for example, use one or more cameras, a lidar, a radar or other sensor mechanisms.

[0004] The computing system is configured to combine objects detected by a plurality of sensors into an object list by means of a first program. Such a program, which for example combines objects into a so-called world view, can be used as a package at least in some configurations.

[0005] The objects are displayed in an object list, wherein each element of the object list includes the position, speed and free distance for each of the objects, and the object list includes a time stamp. These specifications can be recorded, for example, in a memory of the computing system; in this case, these specifications correspond to specifications in the real world. Thus, the spatial coordinates can correspond, for example, to GPS coordinates or to the (if necessary normalized) distance to a vehicle, for example to a vehicle in or on which the sensor is arranged. In one specific embodiment, the positions are represented as spatial coordinates or polar coordinates.

[0006] The speed is a vector which in several embodiments is presented in the computing system in a manner similar to spatial coordinates. A free section can be represented, for example, by two spatial coordinates (as a distance to adjacent static and / or dynamic objects). The free section can be part of a description of the surroundings which can be stored in a memory which the computing system can access. Thus, there is a sequence of object lists as follows: each of these object lists includes a timestamp. In one embodiment, the timestamps can be equidistant, and in another embodiment, the frequency with which the time lists are created can depend on factors of the surroundings - for example, on the number of dynamic objects.

[0007] Furthermore, the computing system is configured to determine, based on a predefined number of object lists, another object list for at least some of the dynamic objects with the aid of a second program, wherein the other object list includes a time stamp for a future moment and at least includes the position of the dynamic objects. Thus, another object list or a predefined number of additional object lists is generated. This is done based on a predefined number of object lists that have already been created in the past; object lists can also be created now. The object list is predicted using the second program, i.e., a list of objects is generated that determines the properties of the objects in the future. For this purpose, for example, linear or nonlinear prediction methods can be used. By means of this arrangement, the prediction of the position of dynamic objects (e.g., pedestrians) that can be detected, for example, by sensors of the vehicle is significantly improved (i.e., accelerated and / or accurately described).

[0008] Another aspect of the invention relates to a method for an at least partially autonomously driven vehicle. The method is used to predict the position of dynamic objects in the surroundings of the vehicle. The dynamic objects may be, for example, other vehicles, pedestrians, cyclists or other traffic participants, in particular traffic participants whose movements may deviate significantly from a straight line or uniform movement.

[0009] The method comprises the following steps:

[0010] -Detect objects with the help of multiple sensors;

[0011] - combining the objects into an object list by means of a first program, wherein each element of the object list comprises a position, a speed and a free path for each of the objects, and

[0012] The object list includes a timestamp;

[0013] - Based on the predefined number of object lists, a further object list is determined for at least some of the dynamic objects by means of a second program, wherein the further object list comprises a time stamp for a future time and comprises at least the position of the dynamic object.

[0014] The sensor can use, for example, one or more cameras, lidars, radars or other sensor mechanisms. Thus, the sensor detects dynamic objects, for example, by means of a single sensor or also by means of a combination of sensors, wherein, for example, different types of sensors can also be combined - for example, the signal of a camera is combined with the signal of a radar sensor. The first program combines the objects detected by multiple sensors into an object list. Here, each element of the object list includes a position, a speed and a free section for each of the objects. In addition, the object list includes a timestamp. The information corresponds, for example, to information in the real world. Therefore, the spatial coordinates can correspond, for example, to GPS coordinates or to a vehicle, for example to a (normalized if necessary) distance to the vehicle in which the sensor is arranged or on the vehicle. In one embodiment, the position is expressed as a spatial coordinate or a polar coordinate.

[0015] Therefore, one of the elements of the object list can correspond to a bicycle (especially a moving bicycle) in the real world. The bicycle is in a certain position at a certain moment. Here, the determined moment corresponds, for example, to the first time stamp of the object list. The position can be presented in the object list, for example, by spatial coordinates. The bicycle has a speed; the speed can be presented in the object list, for example, by a vector of spatial coordinates. In addition, the bicycle has a free section, which can be presented as an end point, for example. In a similar manner, for example, the position, speed and free section of another object (for example a pedestrian) can also be determined and entered into the object list. In the example, the element can be presented in the same object list and can also be attached with a first time stamp.

[0016] Based on the data of this object list and / or the data of other object lists that have been created at an earlier time and are attached with another time stamp, for example, another object list is determined for at least part of these dynamic objects, for example, for bicycles and pedestrians in the example. This other object list includes a time stamp for a future time and at least includes the position of the dynamic objects. Here, the objects can move linearly or non-linearly, for example. Therefore, for example, it can be calculated based on the object list: whether a bicycle or a pedestrian will collide with the vehicle at a future time. The vehicle can be controlled with the help of this information. In the case of a possible collision with the vehicle calculated, the vehicle can, for example, influence the braking system and / or the steering system, that is, for example, deflect from the position of potential collision or brake.

[0017] In one embodiment, the second program uses the so-called LSTM method; this abbreviation stands for the so-called "Long Short-Term Memory" method. This involves a recursive neural network. The recursive neural network is used to process sequential data, especially time-sequential data, that is, in this case, a recursive neural network is used to determine rules and values ​​for another object list, which at least includes the positions of dynamic objects. Here, the LSTM structure is based on an abstraction of the data on which it was trained. This is used in an advantageous manner to model the interaction of different detected objects. Thus, for example, non-linear movements can be detected. In the above example, for example, the following situation can be identified: in the case, a bicycle avoids a pedestrian and this (with the help of the determination of another object list) may lead to a potential collision, which may cause the vehicle to move.

[0018] In one embodiment, the second program uses so-called "social LSTM networks". In this configuration, the behavior of humans in a crowd is particularly taken into account. Advantageously, social LSTM networks use the principle of locality, that is, the objects are particularly likely to be influenced by objects in their surroundings, especially in their immediate surroundings. In the example given here, the probability that a bicycle will react to the movement of a pedestrian is higher than that of another distant object.

[0019] In this case, in the event of a violation of the assumption (i.e. being influenced by objects in its surroundings), a fallback model is used. In this case, the social LSTM network models each of the objects individually. In this case, in one embodiment, a grid is formed around the vehicle and objects in the same grid area (or the same cell of the grid) interact with each other. Thus, a kind of "communication" between the objects (e.g. pedestrians avoiding each other or bicycles) is modeled. In this case, this "communication" between the objects is used, for example, in an advantageous manner.

[0020] The prediction of the position of a dynamic object can thus be significantly further improved. In one embodiment, an artificial neural network is trained with the aid of this information; since, compared to other methods (e.g., rule-based detection of possible predictions of the position), the most important, most frequent and / or most critical situations can be preferably identified based on examples, and the predictions and / or reactions of the control system can thus be significantly improved.

[0021] In one specific embodiment, each element of the object list also includes a probability. This can further increase the reaction speed of the vehicle, since elements with a lower probability can be processed differently from elements with a higher probability. Thus, for example, elements with a higher probability can be processed with a greater prediction depth, or in some computing systems, elements with a higher probability can be processed earlier and / or with a higher priority, or the processing of elements with a higher probability can be interrupted.

[0022] In one embodiment, at least for a portion of the objects, the positions are represented as a bivariate normal distribution. When the size of the object is known, in many cases the exact size is entered into the object list. For example, in the case where the size of the object is unknown, a bivariate normal distribution may be entered instead.

[0023] In one embodiment, the distance is represented in logarithmic form at least for some of the objects. This is particularly advantageous if the represented distances vary greatly for different objects or if objects are further away, for example in the case of a deeper prediction. In one embodiment, a logarithmic representation can be used depending on the distance of the object. In some cases, a lower precision can be used in the case of a logarithmic representation. This is particularly useful for vehicle behavior in different surroundings at different speeds.

[0024] In one specific embodiment, the time stamps have a spacing of 10 ms to 500 ms, in particular 50 ms to 200 ms. The spacing of the time stamps depends primarily on the speed of the host vehicle.

[0025] In one embodiment, the list of objects is a tensor. This has proven to be a particularly interesting method for representation for a range of scenarios.

[0026] A further aspect of the invention relates to a program element which is configured to carry out the method described herein when it is executed on a computing system.

[0027] A further aspect of the invention relates to a computer-readable medium on which the aforementioned program element is stored. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings show:

[0029] Figure 1 A schematic diagram showing an object list according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram showing a sequence of object lists according to one embodiment of the present invention;

[0031] Figure 3A schematic diagram is shown of a method for predicting a position of a dynamic object according to one embodiment of the present invention. DETAILED DESCRIPTION

[0032] Figure 1 2 shows a schematic diagram of an object list 22. In this case, the individual dynamic objects 20 are shown as small cubes and are identified by means of a xyz In this diagram, the x-axis and y-axis are the position coordinates of the object; the z-axis represents the properties of the object. For example, the object a shown in the lower right corner 651 represents an object: the object is at position x=6 and y=5, the object has a first attribute. In the example shown, the number of attributes for each object is the same; however, in other embodiments, a variable number of attributes is possible. In the example shown, the lightly-depicted object a 111 and a 121 is the detected or identified strange object, a 111 For example, it can represent pedestrians, and a 121 For example, it can represent a person riding a bicycle. 341 It can be the own vehicle. Therefore, when spatial coordinates are used as in the embodiment, each of the objects is assigned an xy position. The position can be represented as a bivariate normal distribution. The measurement of the xy position can be linear or, at least for a portion of these objects 20, logarithmic. For each of these objects 20, at least one speed and free path of the object is stored in the attribute list. In addition, the object list 22 includes a timestamp; this timestamp is not shown here because it is a characteristic for the entire object list 22.

[0033] Figure 2 A schematic diagram shows a sequence of object lists 22, which enables the determination of a further object list 23. Here, the further object list 23 is a prediction based on the sequence of the object list 22. Therefore, the object list 23 includes a time stamp for a future time instant.

[0034] exist Figure 2 Three layers 29 of such a prediction are shown in FIG. Each of these layers 29 has two object lists 22, which include observations with detected objects and are t or (for another step) by means of X t+1 In each layer 29, the object list 22 and the internal state H t Determine the output C t, and the next object list 23 is determined in each case (as a prediction). Here, before the past predictions (i.e. with an older time stamp) are passed from one layer 29 to the next layer 29, the past predictions can be corrected with the help of the detected values. In the example shown, this is performed on three layers 29, so that the current object list 23 includes "experience" from the past three steps to a certain extent. A second program is used to make such predictions. The second program can use an LSTM method (especially a social LSTM network) method.

[0035] Figure 3 A schematic diagram of a method 50 for predicting the position of a dynamic object 20 is shown. A dynamic object 20 may be, for example, a pedestrian, a cyclist or another traffic participant, in particular a traffic participant whose movement may deviate significantly from a straight line or uniform movement. In step 51, the dynamic object 20 is detected by means of a plurality of sensors 30. The sensors 30 may use, for example, one or more cameras, lidars, radars or other sensor mechanisms.

[0036] In step 52, the objects 20 are combined with the aid of a first program to form an object list 22. Each element of the object list includes a position, a speed and a free path for each of the objects, and the object list 22 includes a time stamp. The time stamp is created in the past or in the present.

[0037] In step 53, one or more further object lists 22 are created by means of the second program from the predefined number of object lists 22. The further object lists 23 created by means of the second program include time stamps for future instants and at least positions of dynamic objects 20.

Claims

1. A control system (10) for predicting the position of a dynamic object (20) in the surroundings of a vehicle, the control system being for an at least partially autonomous vehicle, the control system comprising: a plurality of sensors (30), A computing system (40) configured to: The objects (20) detected by the plurality of sensors (30) are combined into an object list (22) by means of a first program, wherein: Each element of the object list (22) includes a position, a speed and a free segment for each of the objects (20), and the object list (22) includes a timestamp; Based on a predefined number of object lists (22), a further object list (23) is determined for at least some of the dynamic objects (20) by means of a second program, wherein the further object list (23) includes a time stamp for a future time and at least includes the position of the dynamic object (20), The dynamic objects (20) include the following traffic participants: the movement of the traffic participants deviates strongly from a straight-line or uniform movement, wherein there is a sequence of the following object lists: each of these object lists includes a timestamp, wherein the other object list (23) is a prediction based on the sequence of object lists (22), wherein the interaction of different detected dynamic objects in the object list (22) is modeled in the prediction, and the interaction includes avoidance between the detected dynamic objects.

2. A method for predicting the position of a dynamic object (20) in the surroundings of a vehicle, the method being for an at least partially autonomously driven vehicle, the method comprising the following steps: detecting the object (20) by means of a plurality of sensors (30); The objects (20) are combined into an object list (22) by means of a first program, wherein: Each element of the object list (22) includes a position, a speed and a free segment for each of the objects (20), and the object list (22) includes a timestamp; Based on a predefined number of object lists (22), a further object list (23) is determined for at least part of the dynamic objects (20) by means of a second program, wherein the further object list (23) includes a time stamp for a future time and at least includes the position of the dynamic object (20), The dynamic objects (20) include traffic participants whose movements deviate greatly from a straight-line or uniform movement. There is a sequence of object lists, each of which includes a timestamp, wherein the other object list (23) is a prediction based on the sequence of object lists (22), wherein the interaction of different detected objects is modeled in the prediction, the interaction including avoidance between the detected objects.

3. The method according to claim 2, in, The second procedure uses the LSTM method.

4. The method according to claim 2 or 3, in, The second procedure uses a social LSTM network.

5. The method according to claim 2 or 3, in, Each element of the object list (22) also includes a probability.

6. The method according to claim 2 or 3, in, For at least a portion of the objects (20), the locations are represented by a bivariate normal distribution.

7. The method according to claim 2 or 3, in, For at least a portion of the objects (20), the distances are expressed in logarithmic form.

8. The method according to claim 2 or 3, in, The timestamps have a distance of 10 milliseconds to 500 milliseconds.

9. The method according to claim 2 or 3, in, The list of objects (22) is a tensor.

10. The method according to claim 8, wherein: The timestamps have a distance of 50 milliseconds to 200 milliseconds.

11. A computer program product comprising a computer program which is configured to carry out the method according to any one of claims 2 to 10 when the computer program is executed on a computing system (40). 12 . A computer-readable medium having a computer program stored thereon, the computer program being configured to carry out the method according to claim 2 when the computer program is executed on a computing system (40).

13. A device configured to carry out the method according to any one of claims 2 to 10.

Images