Vehicle control device, vehicle, forecasting device, system, method and computer program product

Abstract

The present subject matter relates to a vehicle control device (10) for controlling a first vehicle (15) when a second vehicle (40) is detected at a location (2, 2a) of a given road infrastructure (1) in the vicinity of the first vehicle (15), comprising: a state estimation unit (101); a vehicle interface unit (104) configured to send a request to predict an expected trajectory of the second vehicle (40) together with the estimated state of the second vehicle (40) to a prediction device (20) and to receive an expected trajectory from the prediction device (20).

Description

[0001] The present subject matter relates to a vehicle control device, a vehicle having a vehicle control device, a prediction device, a system, a method, and a computer program product for controlling a first vehicle when a second vehicle is detected in the vicinity of the first vehicle.

[0002] To increase active safety, modern vehicles are equipped with a variety of advanced driver assistance systems (ADAS) that warn the driver in critical situations or provide independent vehicle guidance. Intersections, in particular, are notorious accident hotspots, where many accidents occur when turning onto another road. To cope with such critical situations, an intersection assistant can assist the driver when entering an intersection. To further increase safety in this regard, knowledge of the future trajectories of vehicles (target vehicles) in the vicinity of the driver's own vehicle (own vehicle) is essential.

[0003] In this context, Patent Literature 1 describes a prediction device including an acquisition unit that acquires a movement history of another vehicle from the other vehicle; a prediction unit that predicts a behavior of the other vehicle based on the movement history; and an output unit that outputs driving support information based on the prediction. The acquisition unit acquires the movement history of the other vehicle within a communication range of inter-vehicle communication, and the prediction unit predicts the behavior of the other vehicle within the communication range of the inter-vehicle communication.

[0004] Patent literature 1: US 2018 / 0370530 A1

[0005] However, performing a prediction of the future trajectory of a target vehicle still requires improvement and, for example, intersections may still pose problems for known systems that should be overcome to increase the comfort of a driver of a self-driver vehicle and safety.

[0006] The subject matter described herein addresses the technical problem of improving the prediction of a future trajectory of a target vehicle, preferably without requiring a direct communication link between the vehicles. This problem is solved by the subject matter of the independent claims. Further preferred developments are described in the dependent claims.

[0007] According to the subject matter set out in the appended claims, a vehicle control device, a vehicle with a vehicle control device, a prediction device, a system, a method and a computer program product for controlling a first vehicle (own vehicle) when a second vehicle (target vehicle) is detected at a location of a predetermined road infrastructure in the vicinity of the first vehicle are provided.

[0008] The given road infrastructure may, for example, include all or specifically selected roads in a city and / or a given region, and a location of the given road infrastructure may, for example, be an intersection or a section of a (multi-lane) road.

[0009] In particular, the subject matter disclosed herein improves the driving comfort and safety of occupants of the first vehicle when a second vehicle is present in its vicinity by taking into account the behavior of the second vehicle when the first vehicle is controlled. It may also be possible for more than one second vehicle to be detected in the vicinity of the first vehicle, whose behavior can be taken into account when the first vehicle is controlled. The proximity / surroundings or the like can be defined as the close space around the host vehicle, where the close space can be defined relatively depending on the speed of the host vehicle, certain areas of interest, etc. When defined as absolute values, the proximity can be defined by a radius of a predetermined number of meters around the host vehicle, where the radius can have a size of 10 m, 20 m, 50 m, 100 m, 200 m, or the like.

[0010] The vehicle control device of the first vehicle includes a state estimation unit that estimates a state of the second vehicle using one or more signals from a plurality of sensors mounted on the first vehicle. The plurality of sensors may include, for example, at least one camera, at least one radar sensor, at least one lidar sensor, at least one ultrasonic sensor, at least one inertial measurement unit (IMU), and / or any other sensor suitable for detecting a second vehicle at a location proximate to the first vehicle and providing measurement signals for estimating the state of the second vehicle. The plurality of sensors may be electrically connected to the vehicle control unit for providing the necessary sensor signals to detect the second vehicle and estimate its state.

[0011] Alternatively or additionally, the state estimation unit may receive the estimated state of the second vehicle from an external unit (external / remote state estimation unit), such as a roadside unit with which the host vehicle communicates, and / or an external, remote computer (server) with which the host vehicle can communicate and receive the estimated state. In other words, the state estimation unit may have a reduced configuration in which it mainly comprises one or more communication units for communicating with the external unit that provides the estimated state. The state estimation unit may include the one or more communication units and the sensors explained above, etc.include, even if it can only use one or the other to receive information about the estimated state of the second vehicle, or the state estimation unit can include the one or more communication units and the sensors explained above and use both to receive / estimate the estimated state of the second vehicle. In the latter option, the state estimation unit can, for example, combine the received estimated state information from the external server with the estimated state of the second vehicle derived from its own sensor data to improve its accuracy. The external unit can, for example, estimate the state of the second vehicle based on information from roadside sensors or related information transmitted from the second vehicle or even other vehicles.

[0012] The vehicle control device further includes a vehicle interface unit that automatically sends a request for predicting the expected / future trajectory of the second vehicle, along with the estimated state of the second vehicle, to a prediction device and receives a predicted expected trajectory from the prediction device. In other words, when a second vehicle is detected at a location near the first vehicle by at least one of the plurality of sensors attached thereto, the state of the vehicle is estimated by the state estimation unit of the vehicle control device based on the received sensor signals (or the data / information received from an external unit), and a request for predicting an expected trajectory of the second vehicle is sent to the prediction device through the vehicle interface unit of the vehicle control device.The request is accompanied by the estimated state of the second vehicle, which is then taken into account by the prediction device when calculating the expected trajectory, together with information about a behavior of several vehicles at the location of the second vehicle.

[0013] The location of the second vehicle can be understood as the location of the given road infrastructure where the second vehicle can be detected near the first vehicle. There may be multiple locations of the second vehicle, which may be distributed across the given road infrastructure and may, for example, include multiple intersections and / or road sections.

[0014] The state of the second vehicle, estimated by the state estimation unit of the first vehicle's vehicle control device or by a remote computer, may include, for example, the vehicle's size, vehicle attitude, vehicle class, position, direction of travel, speed, and / or acceleration / deceleration of the second vehicle. Taking these parameters into account when calculating the expected trajectory enables the latter to be predicted with high accuracy.

[0015] For example, when the first vehicle approaches an intersection (as a location of the predetermined infrastructure) where a second vehicle is located in front of the first vehicle (in its vicinity), the vehicle control device of the first vehicle estimates the state of the second vehicle, or receives it if the latter is detected, such as its position, heading / direction, speed and deceleration, or the like, if any, and sends a request for a prediction of an expected trajectory of the second vehicle to the prediction device. At an intersection, an expected trajectory may, for example, be a left turn, a right turn, or a head-on trajectory. Another example of a location where a second vehicle may be detected in the vicinity of the first vehicle (location of the second vehicle) may be a section of a multi-lane road, e.g.a highway or motorway, on which one or more second vehicles may travel alongside the first vehicle. In this case, an expected trajectory of a second vehicle may, for example, be a lane change.

[0016] By knowing the state of the second vehicle when predicting its expected trajectory, the probability of a possible trajectory can be narrowed down and predicted more accurately. Furthermore, the above configuration and its modifications also offer the possibility of predicting trajectories of multiple (second) vehicles traveling through the road infrastructure.

[0017] The vehicle control device further comprises a vehicle control unit that controls a state of the first vehicle based on the predicted expected trajectory received from the prediction device and / or displays the predicted expected trajectory received from the prediction device to a driver of the first vehicle. The state of the first vehicle may, for example, also include the position, direction of travel, speed, and / or acceleration / deceleration thereof. This means that a behavior of the first vehicle is controlled in response to the expected trajectory of the second vehicle, which is predicted on demand by the prediction device. Knowledge of an expected trajectory of the second vehicle makes it possible to proactively control the state of the first vehicle and thus, for example, to avoid a need for harsh braking or even a collision.

[0018] According to one example, the vehicle control device may further comprise an environment estimation / information unit configured to estimate information about an environment of the second vehicle (environment information) using one or more signals from the plurality of sensors mounted on the first vehicle. The information about the environment of the second vehicle may, for example, include information about traffic and road infrastructure in the environment of the second vehicle, such as traffic in lanes adjacent to the second vehicle and / or traffic signs that regulate the operation of the second vehicle. This information may, for example, be determined using the at least one camera, which may capture one or more images of the environment of the second vehicle to collect information about its environment. Furthermore, the environment information may include map information about the location of the second vehicle, such as, for example,a lane map with, for example, a three-dimensional map with precise positions of some important features of a road, such as lane markings and traffic signs. In this way, the current options of the second vehicle to select a particular direction / route / trajectory can be monitored / estimated by the environment estimation unit of the vehicle control device. Alternatively or additionally, the environment estimation unit can be located remotely from the host vehicle so that a remote computer can perform the environment estimation, and / or instead of sensors of the host vehicle, data / signals / information from external units (collectively referred to as external / remote environment information unit(s)) can be used for the environment estimation, such as map information from an external map provider / computer, such as signals from roadside units, or the like.

[0019] According to one example, the vehicle interface unit of the vehicle control device can send the estimated information about the surroundings of the second vehicle to the prediction device when the request to predict the expected trajectory of the second vehicle is sent. This means that the surroundings information can also be used by the prediction device to narrow down the possible route options of the second vehicle and predict the expected trajectory with high accuracy. Furthermore, the fact that the prediction device receives surroundings information of the second vehicle from the first vehicle, which is monitoring the latter, allows any position of the prediction device and offers the possibility of predicting trajectories of multiple vehicles traveling through a road infrastructure.

[0020] According to one example, the vehicle interface unit of the vehicle control device may be configured to display information about the received predicted expected trajectory of the second vehicle on a display. The display may, for example, be part of the vehicle interface unit and may be mounted within the vehicle in the driver's field of view. The information about the received predicted expected trajectory of the second vehicle may, in addition to the trajectory itself, include, for example, a predicted probability of the expected trajectory, a recommended speed for the first vehicle, and an action indication to the driver. In the case of an autonomously driving vehicle, the recommended speed for the first vehicle and the indication to the driver may merely serve to inform the driver about the manner in which the vehicle is being controlled, with the driver having the option to override the automatic vehicle control.In the case of non-autonomous driving, the recommended speed for the first vehicle and the indication to the driver may be a requirement for the driver to navigate the first vehicle depending on the predicted expected trajectory of the second vehicle.

[0021] Furthermore, a vehicle is provided with a vehicle control device as described above and a plurality of sensors mounted on the vehicle and connected to the vehicle control device. The vehicle may be an autonomously driving vehicle or a driver-driven vehicle with one or more advanced driver assistance systems (ADAS), such as a brake assist system, an intersection assist system, and / or a lane change assist system. In particular, the plurality of sensors may comprise, for example, at least one camera, at least one radar sensor, at least one lidar sensor, at least one ultrasonic sensor, at least one inertial measurement unit (IMU), and / or any other sensor suitable for detecting one or more other vehicles in the vicinity of the vehicle and providing measurement signals for estimating a state of the one or more other vehicles.In particular, a combination of the multiple sensors may be used to estimate the state of the one or more other vehicles, such as their position, direction of travel, speed, and / or acceleration / deceleration, with high accuracy.

[0022] A prediction device for predicting an expected trajectory of a second vehicle (target vehicle) detected at a location of a predetermined infrastructure near a first vehicle (own vehicle) comprises a prediction interface unit that receives a request for predicting an expected trajectory of the second vehicle along with an estimated state of the second vehicle from a vehicle control device of the first vehicle. Furthermore, the prediction interface unit of the prediction device returns a predicted expected trajectory of the second vehicle to the vehicle control device of the first vehicle.

[0023] The prediction device may be a separate server / computer / service cloud located anywhere. It may be located close to the given road infrastructure, or it may be located completely remotely in a server center or the like. In this case, the prediction device may be connected to the vehicle control device via a network such as a wireless local area network, a cellular network, and / or any other network suitable for exchanging data between the prediction device and the vehicle control unit. Alternatively, the prediction device may be included in the vehicle control device of the first vehicle. The estimated state of the second vehicle received by the prediction interface unit of the prediction device may include the position, heading, speed, and / or acceleration / deceleration of the second vehicle.By providing the state of the second vehicle to the prediction device, the latter can be enabled to narrow down the possible route options / trajectories of the second vehicle and predict the expected trajectory with high accuracy. Furthermore, it may be an option for the prediction device and, if located outside the host vehicle, the state estimation unit to be located on the same server / remote computer or on different ones.

[0024] The prediction device further comprises a model unit that predicts the expected trajectory of the second vehicle using a prediction model, taking into account the information about the previous behavior of multiple vehicles at the location of the second vehicle and the received estimated state of the second vehicle. The location of the second vehicle may be, for example, an intersection or a lane of a (multi-lane) road in the given road infrastructure. There may be multiple locations of the second vehicle in the given road infrastructure, which may be, for example, intersections or lanes of a (multi-lane) road. In particular, the location of the second vehicle may be at least one intersection, and the prediction unit may collect information about the behavior of multiple vehicles at the at least one intersection by monitoring a predetermined number of vehicles passing through the latter.

[0025] The behavior information of each monitored vehicle can be stored as a data set in a memory of the prediction device. The prediction model of the model unit, which may preferably be a statistical model, can then predict the expected trajectory based on the collected information about the behavior of the monitored vehicles crossing the intersection.

[0026] The predetermined number of monitored vehicles at the location of the second vehicle may, for example, be a number sufficient to determine a probability distribution regarding the behavior of the monitored vehicles. Specifically, the behavior of the plurality of vehicles passing the location of the second vehicle may be continuously monitored, and if the number of behavior information records exceeds the predetermined number by one, the first record stored in the memory of the prediction device may be deleted, and the last monitored behavior may be stored as a new record in the memory. In this way, the stored records can be kept up-to-date.

[0027] According to one example, the prediction interface unit of the prediction device may be connected to one or more roadside units located at the location of the second vehicle that capture a trajectory and a state of each of a plurality of vehicles thereat, which may be stored together as a data set in the memory of the prediction device.

[0028] In this case, the model unit of the prediction device can determine information about the behavior of the plurality of vehicles at the location of the second vehicle based on the trajectory and state of each of the plurality of vehicles detected by the one or more roadside units. In particular, the state of each vehicle can be assigned to its trajectory when it is stored, so that a behavior of a respective vehicle can also be determined in light of its state. For example, when a vehicle enters an intersection at a speed exceeding a certain value, it can be observed more frequently that the respective vehicle does not turn left or right, but travels straight ahead.

[0029] The one or more roadside units may, for example, be positioned at each intersection of the predefined road infrastructure and / or along one or more roads thereof. Each of the one or more roadside units may include at least one sensor, such as a camera, a radar sensor, a lidar sensor, an ultrasonic sensor, or any other sensor suitable for detecting a trajectory, condition, and surrounding information of each of the multiple vehicles. Furthermore, it may be possible for the prediction device to also be included in one of the one or more roadside units.

[0030] According to one example, the model unit of the prediction device may determine a probability distribution of the detected trajectories at the location of the second vehicle as information about the behavior of the multiple vehicles at the location. To calculate the probability distribution, the model unit may count the number of vehicles that select a specific trajectory at the location of the second vehicle and relate the respective counted number to the total number of vehicles passing the location of the second vehicle. In this regard, the model unit may consider the state of the multiple vehicles at the location of the second vehicle when determining the probability distribution.For example, several probability distributions of the detected trajectories can be determined depending on a speed, a direction of travel and / or an acceleration / deceleration of each vehicle entering / passing through the location of the second vehicle.

[0031] According to one example, the prediction model may be an AI model, and the modeling unit may use the determined information about the behavior of the plurality of vehicles at the location of the second vehicle as training data to train the prediction model. The prediction model trained in this way may then predict the expected trajectory at the location of the second vehicle. The AI ​​model may, for example, be a neural network model that can be trained using the stored data sets containing trajectories and states of the plurality of vehicles detected by the one or more roadside units and / or by the first vehicle at the location of the second vehicle.

[0032] According to one example, the model unit of the predictive model may include a training unit that can receive the training data and label the received training data according to the state of each of the plurality of vehicles at the location of the second vehicle. In particular, the training unit may label the detected trajectories contained in the training data with the states of the corresponding vehicles. In this way, the training unit may provide pre-sorted training data for training the predictive model, which may improve the accuracy and capabilities of the trained AI model.

[0033] According to one example, the prediction interface unit of the prediction device can additionally receive information about an environment of the second vehicle, and the model unit of the prediction device can determine all potential trajectories of the second vehicle based on the received environment information to predict the expected trajectory. The information about the environment of the second vehicle (environment information) can include, for example, information about traffic and road infrastructure in the environment of the second vehicle, such as traffic in lanes adjacent to the second vehicle and / or traffic signs that regulate the operation of the second vehicle. Furthermore, the environment information can include map information about the location of the second vehicle, such as a lane map including, for example, a three-dimensional map with precise positions of some important features of a road, such as lane markings and traffic signs.Such information can be used to determine whether the second vehicle has any chance of turning in one direction or the other or changing lanes. For example, if there is very heavy traffic in an adjacent lane, it can be assumed that the second vehicle does not have the opportunity to turn into that lane. In other words, the information about the second vehicle's surroundings can be used by the model unit to determine all possible options for the second vehicle to select a specific direction / route / trajectory (potential trajectories) and to exclude impossible options. In this way, the efficiency of the predictive model can be increased, since trajectories that are deemed impossible by the model unit in advance can be completely ignored.

[0034] According to one example, the model unit of the prediction device can predict the expected trajectory by determining a probability for each potential trajectory of the second vehicle and selecting the potential trajectory with the highest probability. Specifically, the probability of each potential trajectory of the second vehicle can be determined by the prediction model of the model unit based on the determined probability distributions of the detected trajectories at the location of the second vehicle, given the received state thereof. This means that the model unit can, for example, compare a speed and a direction of travel of the second vehicle, which describe its current state, with a speed and a direction of travel assigned to the different probability distributions that are determined in advance.For example, the model unit can then select a probability distribution labeled with the same or at least a similar speed and direction of travel to determine the probability for each potential trajectory of the second vehicle. Finally, the model unit can select the trajectory with the highest probability as the expected trajectory. If the prediction model is an AI model trained with the determined probability distributions of the detected trajectories at the location of the second vehicle, the AI ​​model can automatically select the probabilities of each potential trajectory of the second vehicle according to the received state of the second vehicle.

[0035] According to one example, the prediction interface unit of the prediction device can receive the state of the second vehicle from at least one of the one or more roadside units. This means that the prediction interface unit can receive the state of the second vehicle not only from the vehicle control unit of the first vehicle, but also from at least one of the roadside units at the location of the second vehicle. This allows for improving the accuracy of the state estimation, since each parameter thereof (e.g., position, speed, acceleration) is determined from a different perspective by different sensors.Moreover, if the state of the second vehicle is determined by more than one roadside unit at different locations of the second vehicle, information about a course of a route of the second vehicle may also be included in the estimated state provided by the respective roadside units.

[0036] According to one example, the prediction interface unit may receive the additional information about the environment of the second vehicle (environment information) from the vehicle control device of the first vehicle and / or from at least one of the one or more roadside units. When the prediction interface unit receives the environment information from the vehicle control device of the first vehicle, a plurality of sensors mounted on the first vehicle may be used to determine the information. The plurality of sensors may include, for example, at least one camera, at least one radar sensor, at least one lidar sensor, at least one ultrasonic sensor, at least one inertial measurement unit (IMU), and / or any other sensor suitable for determining an environment of a second vehicle in the vicinity of the first vehicle.The at least one camera may, for example, capture one or more images of the surroundings of the second vehicle, for example to collect information about traffic in one or more lanes adjacent to a lane in which the second vehicle is present.

[0037] If the prediction interface unit additionally receives environmental information of the second vehicle from one or more roadside units, such information may include the same or different information than the environmental information received from the controller of the first vehicle. For example, the first vehicle may be positioned behind the second vehicle, while the one or more roadside units may be positioned adjacent to and / or in front of the second vehicle. Based on the different positions of the first vehicle and the one or more roadside units, for example, a camera of the first vehicle and a camera of a roadside unit may capture images from different directions and thus provide different environmental information of the second vehicle.In view of this, it is preferable to receive environmental information from both the control device of the first vehicle and the one or more roadside units to ensure that all of the current options of the second vehicle for selecting a particular direction / route / trajectory are determined and to predict the expected trajectory of the second vehicle with high accuracy.

[0038] A system for controlling a first vehicle when a second vehicle is detected at a location of a given road infrastructure near the first vehicle comprises a vehicle control device and a prediction device as described above, and a plurality of roadside units positioned at a plurality of different locations in the given road infrastructure. Each of the plurality of different locations can be a location of the second vehicle, meaning that the second vehicle can be detected at these locations near the first vehicle. The plurality of roadside units can, for example, be positioned at a plurality of intersections and / or road sections of the given road infrastructure. The prediction device can be a separate server / computer / service cloud arranged in an arbitrary position. In this case, the prediction device can be connected to the vehicle control device via a network such as, for example,a wireless local area network, a cellular network, and / or any other network suitable for exchanging data between the prediction device and the vehicle control unit. Alternatively, the prediction device may be included in the vehicle control device of the first vehicle.

[0039] According to one example, the one or more roadside units of the plurality of roadside units form a roadside unit network configured to exchange information between different roadside units. This allows, for example, a first state of the second vehicle determined by a first roadside unit positioned at a first location in the predetermined road infrastructure to be sent to a second roadside unit that determines a second state of the second vehicle at a second location in the predetermined road infrastructure. Each of the first and second states of the second vehicle may include, for example, a position, a direction of travel, a speed, and / or an acceleration / deceleration of the second vehicle at the first and second locations, respectively.In this way, the second roadside unit can, for example, determine a route of the second vehicle through the given road infrastructure.

[0040] Furthermore, any configuration of the disclosed vehicle control device, prediction device, or system described above is also intended to be encompassed by a method, which may be claimed alone and / or by a computer program product claim.

[0041] In summary, a vehicle control device, a prediction device, a system, a method, and a computer program product for controlling a first vehicle (own vehicle) when a second vehicle (target vehicle) is detected in its vicinity can be created, which in particular increase the safety of occupants of the first vehicle by taking into account an expected behavior of the second vehicle when the first vehicle is controlled. The expected behavior of the second vehicle is predicted based on information about the previous behavior of several vehicles monitored at a location of the second vehicle, taking into account a state of each of the several monitored vehicles, which can be, for example, the vehicle type, position, direction of travel, speed, and / or acceleration / deceleration of the respective vehicle. The information about the vehicle type or vehicle class, etc.can be used as additional “status” information, if available. Brief description of the drawings Fig. 1 schematically shows an exemplary situation at a location of a given road infrastructure in which the present invention can be used. Fig. 2 shows schematically and by way of example a system according to an embodiment of the present invention, which comprises a vehicle control device, a prediction device and a plurality of roadside units. Fig. Figure 3 shows schematically and by way of example a training unit of the prediction device, which in Fig. 2 is shown. Fig. Figure 4 shows a schematic example of how training data is processed by the Fig. 3 shown training unit can be generated. Fig. Figure 5 shows schematically and exemplarily a result of the Fig. 4 shown training data generation. Fig. 6a and Fig. 6b show schematically another exemplary situation at the location of the given road infrastructure, which in Fig. 1 in which the present invention can be used. Fig. 7a and Fig. 7b schematically show another exemplary situation at the location of the given road infrastructure, which in Fig. 1 in which the present invention can be used. Fig. 8a and Fig. 8b schematically show another exemplary situation at a different location of the given road infrastructure in which the present invention can be used. Embodiments of the disclosed subject matter

[0042] Embodiments of the present invention are described in detail below with reference to exemplary figures. The features of the embodiments can be combined in whole or in part, and the present invention is not limited to the described embodiments. In the figures, identical elements are provided with identical reference numerals, so a repeated description of the elements is omitted unless necessary.

[0043] Fig. 1 schematically shows an exemplary situation at a location 2 of a given road infrastructure 1 in which the present invention can be used. The given road infrastructure 1 can, for example, comprise all or specifically selected roads 1a-1d in a city or a given region. The illustrated location 2 is an intersection 2 of the given infrastructure 1, into which a target vehicle 40 (second vehicle) enters from a first road 1a. The intersection 2 allows driving straight ahead into a second road 1b, turning left into a third road 1c, and turning right into a fourth road 1d. Thus, there are three potential trajectories that the target vehicle can choose, with the fourth road 1d leading to a construction site 50 that cannot be driven through.

[0044] Furthermore, a private vehicle 15 (first vehicle) is in Fig. 1, which is located behind the target vehicle and approaching the intersection 2 at a certain distance from the target vehicle 40. The host vehicle 15 may be an autonomous vehicle or a driver-driven vehicle with one or more advanced driver assistance systems (ADAS) such as a brake assist system, an intersection assist system, and / or a lane change assist system, and may be equipped with a plurality of sensors 100, which may include at least one camera, at least one radar sensor, at least one lidar sensor, at least one ultrasonic sensor, and / or at least one inertial measurement unit (IMU) (not shown in detail). The plurality of sensors may enable the host vehicle 15 to detect the target vehicle 40 traveling nearby, namely in front of the host vehicle 15, and to determine a status (position, direction of travel, speed, acceleration / deceleration, etc.) of the target vehicle 40.) and environmental information of the target vehicle 40.

[0045] Two roadside units 30a, 30b are arranged along the first road 1a, with a first roadside unit 30a monitoring the first road 1a at a distance from the intersection 2, and a second roadside unit 30b monitoring the location where the first road enters the intersection 2. A third roadside unit 30c is arranged adjacent to the third road 1c, and a fourth roadside unit 30d is arranged adjacent to the fourth road 1d, both monitoring the respective roads 1c, 1d at a short distance from the intersection 2. In addition, a fifth roadside unit 30e is arranged along the second road 1b at a distance from the intersection 2.

[0046] Each of the exemplary five roadside units 30a-30e may include at least one sensor, such as a camera, a radar sensor, a lidar sensor, and / or an ultrasonic sensor (not shown). The at least one sensor may enable the roadside units 30a-30e to capture a trajectory, condition, and surrounding information of each vehicle passing on any of the four roads 1a-1d.

[0047] Specifically, the first, second, and fifth roadside units 30a, 30b, 30e can be used to detect a trajectory of each vehicle approaching intersection 2 from road 1a and traveling straight through intersection 2 into road 1b. Similarly, the trajectory of a vehicle turning left at intersection 2 into road 1c and a vehicle turning right into road 1d can be detected using the second and third roadside units 30b, 30c and the second and fourth roadside units 30b, 30d, respectively.

[0048] Information about the surroundings of a vehicle (surrounding information) may include information about traffic and road infrastructure in the vicinity of a vehicle on any of the four roads 1a-1d, such as traffic in lanes adjacent to the vehicle and / or traffic signs regulating the operation of the vehicle at the intersection 2 (not shown). Such information may be detected, for example, by a camera included in each of the illustrated roadside units 30a-30e (not shown in detail) and / or by any of the sensors of the host vehicle. Furthermore, the surrounding information may include map information about a location of a vehicle, such as a lane map of the intersection 2 with precise positions of some important features thereof, such as lane markings and traffic signs, in a three-dimensional view.

[0049] In the Fig. In the situation illustrated in Figure 1, it may be helpful for a driver, and in the case of autonomous driving, for a control unit of the host vehicle 15, to have information about an expected trajectory of the target vehicle 40 entering the intersection 2 in order to decide how the host vehicle 15 should be controlled, for example, with regard to its speed and direction of travel. In the illustrated example, the detected trajectory, the state, and environmental information of each vehicle passing the roadside units 30a-30e on one of the four roads 1a-1d can be used to calculate at least one probability distribution that provides, for each of the possible roads 1b-1d, a probability that a vehicle entering the intersection 2 will select this road (indicated by the question marks at the arrows indicating the three potential trajectories). The probability distribution can be determined by a prediction device (in Fig. 1 (not shown), which may be included, for example, in one of the roadside units 30a-30e. It may also be possible for the prediction device to be a separate server / computer / service cloud arranged in any desired location. Alternatively, the prediction device may be included in a vehicle control device of the host vehicle 15. In particular, a plurality of probability distributions may be determined by the prediction device based on the detected trajectories and the corresponding conditions and environmental information.

[0050] When the target vehicle 40 is detected, a vehicle control device (in Fig. 1) of the host vehicle may send a request to predict an expected trajectory of the target vehicle 40 along with an estimated state thereof to the prediction device. The vehicle control device of the host vehicle may estimate the state of the target vehicle 40 based on measurement signals from the plurality of sensors 100 mounted on the host vehicle 15 and / or based on information received from an external unit.

[0051] It may also be possible for the host vehicle to transmit environmental information of the target vehicle 40, which is detected by at least one of the plurality of sensors 100, to the prediction device. Alternatively or additionally, at least one of the five roadside units 30a-30e can transmit the state of the target vehicle 40 and / or environmental information thereof to the prediction device.

[0052] The prediction device may calculate a probability for each of the three potential trajectories and may select the trajectory with the highest probability as the expected trajectory of the target vehicle 40. The expected trajectory, as well as the probabilities for each of the three potential trajectories, may be sent to the vehicle control device of the host vehicle to enable improved control thereof, taking into account the behavior of the target vehicle 40.

[0053] Fig. Figure 2 shows schematically and by way of example a system according to an embodiment of the present invention, which comprises a vehicle control device 10, a prediction device 20 and the five roadside units 30a - 30e already described in Fig. 1, which form a roadside unit network 30.

[0054] By forming a network 30, the various roadside units 30a-30e can exchange information with each other. For example, the first roadside unit 30a, positioned along the first road 1a, can determine a first state of the target vehicle 40 passing by, and the second roadside unit 1b can determine a second state of the target vehicle 40 as it enters the intersection 2 (see Fig. 1). When the second roadside unit 30b detects the target vehicle 40, it can send a request to any of the other roadside units 30a-30e to check whether at least one of them has previously detected the target vehicle 40. In this way, for example, a course of a route of the target vehicle 40 through the given road infrastructure 1 can be determined by the roadside unit network 30, which can be included in the state of the target vehicle 40 and updated at each roadside unit 30a-30e.

[0055] The roadside unit network 30 is connected to the prediction device 20, which in the example shown comprises a prediction interface unit 202, a target vehicle state unit 200, an environment information unit, and a model unit 210. The model unit 210 in turn comprises a prediction model 211, which may in particular be an AI model, and a training unit 212 (the training unit may not be present after the AI ​​model has been trained prior to its deployment). The prediction device 20 may be a separate server / computer / service cloud located at any position in the given road infrastructure 1 or elsewhere. In this case, the prediction device 20 may be connected to the vehicle control device 10 via a network such as, for example,a wireless local area network, a cellular network, and / or any other network suitable for exchanging data between the prediction device and the vehicle control unit. Alternatively, the prediction device 20 can be included in the vehicle control device 10 of the host vehicle 15.

[0056] The illustrated roadside units 30a-30e of the roadside unit network 30 can record multiple trajectories of multiple vehicles crossing the intersection 2, along with a state and environmental information of each vehicle, and can send the trajectories thus recorded to the training unit 212 of the model unit, which can use the received trajectories to train the prediction model 211. The training unit 212 can, for example, calculate a probability distribution based on the received trajectories that provides, for each of the possible roads 1b-1d, a probability that a vehicle entering the intersection 2 will select that road. In particular, the training unit 212 can label each received trajectory with its state and environmental information and can calculate multiple probability distributions for different states and environmental conditions of a target vehicle 40.In this way, the training unit 212 may provide pre-sorted training data for training the prediction model, which may improve the accuracy and capabilities of the trained prediction model 211.

[0057] The Fig. The vehicle control device 10 shown in Fig. 2 can be attached to the own vehicle 15 and comprises a vehicle control unit 103, a vehicle interface unit 104 as well as a state estimation unit 101 and an environment estimation unit 102, both of which measure signals from the plurality of sensors 100 mounted on the own vehicle 15 (see Fig. 1) and / or received from external units.

[0058] When the target vehicle 40 located in the intersection 2 is detected by at least one of the plurality of sensors 100 of the host vehicle 15 connected to the vehicle control device 10, the state estimation unit 101 and the surrounding environment estimation unit 102 can determine / estimate a state and surrounding environment information of the target vehicle 40 using measurement signals from the plurality of sensors 100 and / or using information received from roadside units 30a-30e or other information sources. For example, the state estimation unit 101 can determine a position of the target vehicle 40 in the intersection 2 relative to the position of the host vehicle 15, as well as its speed and acceleration, using a radar sensor, a lidar sensor, and / or a camera mounted on the host vehicle 15.Alternatively or additionally, one or more of the roadside units 30a-30e may transmit sensor signals via the second vehicle 40 to a remote computer in which the state estimation unit 101 is arranged (in addition or alternatively to arranging it in the control device of the host vehicle) and which remotely determines the state of the second vehicle 40 and transmits the estimated state to the host vehicle 15.

[0059] The environment estimation unit 102 can, for example, estimate information about infrastructure and traffic in the surroundings of the target vehicle 40 using the plurality of sensors 100. In the Fig. In the situation shown in Figure 1, the environment estimation unit may, for example, determine that road 1d is frequently avoided by other vehicles, e.g., due to a road closure or the like. It is not necessary for the environment information unit 102 to actually detect the specific cause of the other vehicles' behavior. Alternatively, the environment information unit 102 may detect the cause, e.g., that the road leads to a construction site 50 that cannot be driven through. In this regard, the environment estimation unit may, for example, receive a signal from a camera mounted on the host vehicle 15, which may, for example, have captured an image of a traffic sign informing about the construction site 50.

[0060] To control the host vehicle 15 in light of the behavior of the target vehicle 40, the vehicle control device 10 of the host vehicle 15 may send a request to predict an expected trajectory of the target vehicle 40, along with the estimated state and surrounding information, to the prediction device 20 via the vehicle interface unit 104. The prediction device 20 may further receive a current state and current surrounding information of the target vehicle 40 from the roadside unit network 30. The received target vehicle state information may be combined in the target vehicle state unit 200, and the received surrounding information may be combined in the surrounding information unit 201 of the prediction device 20 before both are forwarded to the model unit 210.The model unit 210 can predict the expected trajectory of the target vehicle 40 using the prediction model 211 trained with the multiple trajectories detected by the roadside units 30a-30e and / or the host vehicle 15, taking into account the received state and surrounding information of the target vehicle 40. Specifically, the prediction model 211 can calculate a probability for each of the roads 1b-1d that the target vehicle 40 can select (potential trajectories), and can determine a trajectory into the road with the highest probability as the expected trajectory.

[0061] The expected trajectory, as well as the probabilities for each of the three potential trajectories, can be sent from the model unit 210 via the prediction interface unit 202 of the prediction device 20 to the vehicle interface unit 104 of the vehicle control device 10. Furthermore, the prediction device 20 can calculate driving instructions for the host vehicle 15, such as a recommended speed and a driver warning, which can also be sent to the vehicle interface unit 104. The vehicle interface unit 104 can display the received information on the display 104a and / or transmit it to the vehicle control unit 103.If the host vehicle 15 is an autonomous vehicle, the recommended speed and the driver notification may merely serve to inform the driver about the manner in which the host vehicle 15 is controlled, with the driver having the option to override the automatic vehicle control. In the case of non-autonomous driving, the recommended speed for the host vehicle 15 and the driver notification may be a requirement for the driver to navigate the host vehicle 15 depending on the predicted expected trajectory of the vehicle. In this way, the host vehicle 15 may be controlled in light of the behavior of the upstream target vehicle 40.

[0062] Fig. Figure 3 shows schematically and by way of example a training unit 212 of the prediction device, which in Fig. 2. The training unit 212 includes a data designation unit 2120, a memory 2121, and a training module 2122.

[0063] As described above, the training unit 212 may receive multiple trajectories of multiple vehicles crossing the intersection 2, as well as a state and environmental information of each vehicle, from the roadside unit network 30. The data designation unit 2120 may designate each received trajectory with its state and environmental information and may transfer the designated trajectories to the storage 2121. The training module 2122 may cluster the designated trajectories according to their state and environmental information and may calculate multiple probability distributions based on the clustered trajectories. It may also be possible for the designation unit 2120 to already cluster the designated trajectories and calculate the corresponding probability distributions.

[0064] Each cluster may include lanes whose state lies within a predetermined range. e.g., a speed and / or acceleration / deceleration may lie between a first minimum and maximum value in a first cluster and between a second minimum and maximum value in a second cluster. Likewise, environmental information from lanes contained in the same cluster may also lie within a predetermined range. For example, the number of vehicles present in an adjacent lane may lie between a first minimum and maximum value in a first cluster and between a second minimum and maximum value in a second cluster.

[0065] The clusters of labeled trajectories and the multiple probability distributions can also be stored in memory 2121. The training module 2022 can then train / calibrate the prediction model 211 using the multiple probability distributions of the clustered trajectories. In this way, multiple pre-sorted training data are available for training the prediction model 211, which can improve the accuracy and capabilities of the trained prediction model 211.

[0066] Fig. Figure 4 shows a schematic example of how training data is processed by the Fig. 3. In particular, the example illustrates the generation of training data representing the behavior of several vehicles that have the Fig. 1. For this purpose, the data designation unit 2120 of the training unit 210 may receive a trajectory of each vehicle passing through the intersection, along with a state and environmental information, from the roadside unit network 30. Four trajectories No. 1 to No. 4 of four vehicles 40-1 to 40-4 are shown crossing the intersection 2, where the first trajectory No. 1 represents a right turn, the second and third trajectories No. 2 and No. 3 represent a left turn, and the fourth trajectory represents a straight-ahead crossing of the intersection 2.

[0067] The data designation unit 2120 may designate each received trajectory No. 1 to No. 4 with a state and surrounding information of the corresponding vehicle 40-1 to 40-4, and may transmit the designated trajectories No. 1 to No. 4 to the training module 2122, which may calculate a probability distribution with respect to the intersection 2 each time it receives a new designated trajectory from the designation unit 2120.

[0068] For example, after receiving the first lane No. 1 as the very first lane, the model unit 210 can calculate the probability distribution as 100% right turn, 0% left turn, and 0% straight ahead. When receiving the subsequent lanes No. 2 through No. 4, the model unit 210 can adjust the probability distribution as follows: No. 2: 50% right turn, 50% left turn, 0% straight ahead; No. 3: 33.3% right turn, 66.6% left turn, 0% straight ahead; and No. 4: 25% right turn, 50% left turn, 25% straight ahead.

[0069] In this context, the data designation unit 2120 or the training module 2122 may cluster the designated trajectories depending on their state and environmental information, and the training module 2122 may calculate a probability distribution as described above for each cluster of trajectories as training data 2121a, which is then stored in the memory 2121.

[0070] Fig. Figure 5 shows schematically and exemplarily a result of the training data generation for the intersection 2, which is shown in Fig. 4. In particular, the illustrated training data 2121a comprises a list with three columns, the first column comprising a data ID that identifies the different clusters of trajectories, the second column comprising the state of the vehicles assigned to a respective cluster, and the third column comprising the probability distribution of the trajectories assigned to the respective cluster. The state preferably comprises the parameters location, speed, and orientation (direction of travel) of the vehicles that provide the trajectories for each cluster X1 to Z1. A cluster may comprise states that lie within a predetermined range. In other words, each cluster may contain a minimum and a maximum value for a position, a speed, and an orientation.For example, a position of P=10, a velocity of V=25 and an orientation of 0=0 can belong to the same cluster X1 as a position of P=10.5, a velocity of V=25 and an orientation 0=0.2.

[0071] According to the training data 2121a, as shown, a vehicle located at a position P1 in the intersection 2 and having a speed V1 and an orientation 01 (data ID X1) has a straight-ahead probability of 35%, a right-turn probability of 5%, and a left-turn probability of 60%. Consequently, when the vehicle control device 10 of the host vehicle 15 estimates the vehicle state X1 for the upstream target vehicle 40 and sends it to the prediction device 20 along with a request to predict an expected trajectory of the target vehicle 40, the model unit 210 of the prediction device 20 can predict a left turn as its expected trajectory. If the target vehicle 40 is located at a position P2 in the intersection (data ID Y1) with the same speed and orientation V1, 01, the prediction unit 210 can predict that the target vehicle is traveling straight ahead, since the probability of this is 80%.For example, position P2 may be a position in a center lane of multiple lanes at intersection 2. If the state of target vehicle 40 corresponds to that of the cluster with data ID Z1, model unit 210 of prediction model 20 may predict a 60% probability that target vehicle 40 will travel straight, 30% that it will turn right, and 10% that it will perform a U-turn.

[0072] By performing a training of the prediction model 211 in the training module 2122 using the training data 2121a, as in Fig. 5, which include probability distributions for potential trajectories at the intersection 2 associated with several possible states of the target vehicle 40, a reliable prediction of the expected trajectory can be provided by the prediction device 20.

[0073] Fig. 6a and Fig. 6b show schematically another exemplary situation at the intersection 2 of the Fig. 1 shown predetermined road infrastructure in which the present invention can be used.

[0074] The intersection 2 allows driving straight ahead into a second street 1b, turning left into a third street 1c, and turning right into a fourth street 1d. Consequently, there are three potential paths that the target vehicle 40 can choose, with the fourth street 1d leading to a construction site 50 that cannot be driven through. Fig. 6a and Fig. In the situation shown in Figure 6b, the target vehicle 40 enters the intersection and slows down (indicated by the arrows decreasing in length next to the target vehicle 40).

[0075] The Fig. The host vehicle 15b shown in Figure 6a may include a controller that only evaluates the state of the preceding target vehicle 40 to estimate whether or not it is turning left and to control a possible overtaking maneuver of the host vehicle 15b. This means that the controller does not receive any information from the roadside units 30a-30e about an expected trajectory of the target vehicle (indicated by the crosses above the roadside units and their reference numerals). Consequently, the controller of the host vehicle 15b must estimate an equal probability of 33.3% for each possible trajectory of the preceding target vehicle 40.

[0076] On the other hand, the Fig. 6b, the own vehicle 15a may comprise a vehicle control unit 10, as shown in Fig. 2, which sends a request to a prediction device 20 for receiving an estimated trajectory of the target vehicle 40 based on a current state and environmental information of the target vehicle 40 estimated by the host vehicle 15a and / or the roadside units 30a-30e. As described above, the prediction device 20 may include a model unit 210 having a prediction model 211 trained with multiple trajectories detected by vehicles crossing the intersection 2 and having different states and environmental conditions. Consequently, the prediction model 211 considers the behavior of multiple vehicles that have crossed the intersection while the construction site 50 blocks the passage of the fourth road 1d. Consequently, the prediction device 20 calculates a probability of only 5% that the target vehicle 40 will turn right onto the fourth road 1d leading to the construction site 50.Furthermore, the prediction device 20 calculates a 70% probability that the target vehicle 40 will turn left and a 25% probability that the target vehicle 40 will drive straight. The prediction device 20 can send the probability distribution to the vehicle control device 10 of the host vehicle 15, which can then control the host vehicle 15 so that it does not overtake the target vehicle 40 on the left. From the comparison of . Fig. 6a and Fig. 6b it becomes clear that a combination of the estimation of the state and environmental information of the target vehicle 40 and the consideration of statistical information regarding the behavior of vehicles at the intersection 2 makes it possible to provide a reliable prediction of the expected trajectory of the target vehicle.

[0077] Fig. 7a and Fig. 7b schematically show another exemplary situation at the intersection 2 of the given road infrastructure 1, which in Fig. 1 in which the present invention can be used. The potential trajectories that the target vehicle 40 can choose are the same as in Fig. 1, Fig. 6a and Fig. 6b. In the Fig. 7a and Fig. In the situation shown in Figure 7b, the target vehicle 40 enters the intersection at a reduced speed and positions itself therein with its front facing to the right.

[0078] The Fig. The host vehicle 15c shown in Figure 7a may include a control device that uses only statistical information regarding the behavior of vehicles at the intersection 2 to estimate whether or not to turn left, and does not consider the current state of the target vehicle 40 in front to control a possible overtaking maneuver of the host vehicle 15b. This means that the control device of the host vehicle 15c has no information about the position and speed of the target vehicle 40. In other words, in the situation shown, the host vehicle cannot detect that the target vehicle 40 has positioned itself with its front facing to the right.Therefore, the control device of the host vehicle 15c estimates only a 5% probability that the target vehicle 40 will turn right, since only a few vehicles, especially only construction vehicles, have entered the fourth road 1d in the past due to the construction site 50. Furthermore, the control device of the host vehicle 15c estimates, with only statistical information about the traffic at the intersection 2, a 60% probability that the target vehicle 40 will go straight and a 35% probability that the target vehicle 40 will turn left.

[0079] On the other hand, the Fig. 7b, the own vehicle 15a may comprise a vehicle control unit 10, as shown in Fig. 2, which estimates the current state and environmental conditions of the target vehicle, which are then taken into account by the prediction device when calculating the probability distribution using a trained prediction model. Alternatively or additionally, one or more of the roadside units 30a-30e can transmit sensor signals via the second vehicle 40 to a remote computer in which the state estimation unit 101 is arranged (in addition to or alternatively to arranging the same in the control device of the host vehicle) and which remotely determines the state of the second vehicle 40 and transmits the estimated state to the host vehicle 15a. In the Fig. 7a and Fig. 7b, this means that the prediction device 20 is aware that the target vehicle 40 has already turned its front to the right, even if the road 1d on the right side of the intersection 2 leads to the construction site 50. Consequently, the prediction device 20 calculates a probability of 95% that the target vehicle 40 will turn right onto the fourth road 1d to the construction site 50, and only 3% or 2% that it will choose a different direction. By comparing the predicted probability distributions shown in Fig. 7a and Fig. 7b, it becomes apparent that monitoring the current state of a target vehicle 40 is essential for a reliable prediction of its expected trajectory. In particular, a combination of estimating the state and environmental conditions of the target vehicle and taking into account statistical information regarding the behavior of vehicles at the intersection 2 enables a further improvement in the accuracy of the prediction.

[0080] Fig. 8a and Fig. 8b schematically show another exemplary situation at another intersection 2a of the given road infrastructure 1 in which the present invention can be used. In particular, Fig. 8a and Fig. 8b, such as the vehicle control device 10 shown in Fig. 2, controls the own vehicle 15 in response to a probability distribution of potential trajectories of an oncoming target vehicle 40a, the probability distribution being determined by the prediction device 20 shown in Fig. 2 is calculated.

[0081] As with intersection 2, four roads 1aa, 1ba, 1ca, 1da intersect at intersection 2, and four roadside units 30ba, 30ca, 30da, and 30ea are arranged at the edge of each of roads 1aa, 1ba, 1ca, 1da. A roadside unit 30aa is arranged along a first road 1aa, monitoring the location where the first road 1aa enters intersection 2a. A second roadside unit 30ba is arranged along a second road 1ba, leading straight out of intersection 2a, and two further roadside units 30ca, 30da are located at third and fourth roads 1ca, 1da, leading left and right out of intersection 2a, respectively. The latter monitor vehicles traveling along the respective roads 1ba, 1ca, 1da as they leave the intersection. By combining the information from the four roadside units, trajectories of multiple vehicles crossing intersection 2a can be detected.In contrast to intersection 2, all four streets 1aa to 1da can be driven through, as no construction site blocks any of them.

[0082] In the Fig. 8a and Fig. In the situation shown in Fig. 8b, the vehicle control device 10 of the host vehicle 15 controls the latter to travel straight through the intersection and requests a prediction of an expected trajectory of the oncoming target vehicle 40a from the prediction device for determining a suitable speed for passing the intersection 2a.

[0083] According to the Fig. Based on the probability distribution shown in FIG. 8a (probabilities added to the arrows indicating the potential trajectories of the target vehicle 40a), the prediction device 20 calculates a 60% probability that the target vehicle 40 will turn onto the third road 1c, a 5% probability that it will go straight, and a 35% probability that it will turn onto the fourth road 1da that crosses the path of the host vehicle 15. Due to the relatively high probability that the target vehicle 40a will cross the path of the host vehicle, the vehicle control device 10 can control the host vehicle 15 to enter the intersection indicated by the thin arrow in front of the host vehicle 15 at a low speed (e.g., less than or equal to 15 km / h).The recommended vehicle speed along with an indication to a driver to be aware of a potentially crossing vehicle may be shown on the display 104a of the vehicle control device 10.

[0084] In contrast, the Fig. 8b, calculated by the prediction device 20, a 90% probability that the target vehicle 40 will turn into the third street 1c, a 5% probability that it will go straight, and only a 5% probability that it will turn into the fourth street 1da, which crosses the path of the host vehicle 15. Due to the relatively low probability that the target vehicle 40a will cross the path of the host vehicle, the vehicle control device 10 controls the host vehicle 15 to enter the intersection at a higher speed (e.g., higher than 15 km / h up to 40 km / h), which is indicated by the bold arrow in front of the host vehicle 15. Again, the recommended vehicle speed can be shown on the display 104a of the vehicle control device 10, along with a corresponding indication to the driver. From the two Fig. 8a and Fig.8b, it becomes clear that a reliable prediction of an expected trajectory of the target vehicle, as provided by the above-shown embodiments of the present invention, enables the control of a host vehicle to be improved so that its behavior can be adapted to a behavior of the target vehicle.

[0085] In summary, a vehicle control device, a prediction device, a system, a method, and a computer program product for controlling a first vehicle (own vehicle) when a second vehicle (target vehicle) is detected in its vicinity can be created, which in particular increase the safety of occupants of the first vehicle by taking into account an expected behavior of the second vehicle when controlling the first vehicle. The expected behavior of the second vehicle is predicted based on information about the behavior of multiple vehicles monitored at a location of the second vehicle, taking into account a state of each of the multiple monitored vehicles, which may be, for example, a position, a direction of travel, a speed, and / or an acceleration / deceleration of the respective vehicle.

[0086] It is further noted that examples of the present disclosure may take the form of a complete hardware example, a complete software example (including firmware, resident software, microcode, etc.), or an example that combines software and hardware aspects. Furthermore, examples of the present disclosure may take the form of a computer program product on a computer-readable medium having computer-executable program code embodied in the medium.

[0087] It should be noted that arrows in the drawings may be used to represent communication, transmission, or other activity involving two or more entities. Double-ended arrows generally indicate that the activity can occur in either direction (e.g., a command / request in one direction with a corresponding response back in the other direction, or peer-to-peer communications initiated by an entity), although in some situations the activity may not necessarily occur in either direction.

[0088] Unidirectional arrows can generally indicate activity exclusively or predominantly in one direction, although it should be noted that in certain situations, such directional activity may actually involve activity in both directions (e.g., a message from a sender to a receiver and an acknowledgment back from the receiver to the sender, or establishing a connection before a transmission and terminating the connection after the transmission). Consequently, the type of arrow used in a particular drawing to represent a particular activity is exemplary and should not be seen as limiting.

[0089] Aspects / examples are described above with reference to flowchart illustrations and / or block diagrams of methods and devices, and the like. It should be understood that each block of the flowchart illustrations and / or block diagrams and / or combinations of blocks in the flowchart illustrations and / or block diagrams may be implemented by computer-executable program code.

[0090] The computer-executable program code may be delivered to a processor of a general purpose computer, special purpose computer, or other programmable data processing device to produce a special purpose machine, such that the program code, executing via the processor of the computer or other programmable data processing device, produces means for implementing the functions / acts / outputs specified in the flowchart, block diagram block or blocks, figures, and / or written description.

[0091] This computer-executable program code may also be stored in a computer-readable memory that can instruct a computer or other programmable data processing device to function in a specific manner such that the program code stored in the computer-readable memory produces an article of manufacture having instruction means that implement the function / act / output specified in the flowchart, the one or more block diagram blocks, the figures, and / or the written description.

[0092] The computer-executable program code may also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, such that the program code executing on the computer or other programmable device provides steps for implementing the functions / acts / outputs specified in the flowchart, the one or more block diagram blocks, the figures, and / or the written description. Alternatively, the steps or acts implemented by the computer program may be combined with steps or acts implemented by an operator or human to carry out an embodiment.

[0093] Communications networks may generally include public and / or private networks; may include local, wide area, metropolitan area, storage, and / or other types of networks; and may employ communications technologies including, but not limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), networking technologies, and internetworking technologies.

[0094] It should also be noted that devices may use communication protocols and messages (e.g., messages generated, transmitted, received, stored, and / or processed by the device), and such messages may be conveyed through a communications network or communications medium.

[0095] Unless the context requires otherwise, the present disclosure should not be construed as limited to any particular communication message type, any particular communication message format, or any particular communication protocol. Thus, a communication message may generally include, without limitation, a frame, a packet, a datagram, a user datagram, a cell, or other type of communication message.

[0096] Unless the context requires otherwise, references to particular communication protocols are exemplary, and it should be understood that alternative embodiments may, as appropriate, employ variations of such communication protocols (e.g., modifications or extensions to the protocol that may be made from time to time) or other protocols either known or developed in the future.

[0097] It should also be noted that logic flows may be described herein to demonstrate various aspects and should not be construed as limiting the disclosure to any particular logic flow or logic implementation. The described logic may be divided into various logic blocks (e.g., programs, modules, functions, or subroutines) without changing the overall results.

[0098] Often, logic elements can be added, modified, omitted, performed in a different order, or implemented using other logic constructs (e.g., logic gates, loop primitives, conditional logic, and other logic constructs) without changing the overall results.

[0099] The present disclosure may be embodied in many different forms, including, but not limited to, computer program logic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general-purpose computer), programmable logic for use with a programmable logic device (e.g., a field-programmable gate array (FPGA) or other PLD), discrete components, an integrated circuit assembly (e.g., an application-specific integrated circuit (ASIC)), or any other means, including any combination thereof.Computer program logic that implements some or all of the described functionality is typically implemented as a set of computer program instructions converted into a computer-executable form, stored as such on a computer-readable medium, and executed by a microprocessor under the control of an operating system. Hardware-based logic that implements some or all of the described functionality may be implemented using one or more suitably configured FPGAs.

[0100] Computer program logic that implements all or part of the functionality previously described herein may be embodied in various forms, including, but by no means limited to, source code form, computer executable form, and various intermediate forms (e.g., forms generated by an assembler, compiler, program linker, or locator).

[0101] The source code may comprise a set of computer program instructions implemented in any of various programming languages ​​(e.g., object code, assembly language, or a high-level language such as Fortran, C, C++, Java, or HTML) for use with various operating systems or operating environments. The source code may define and use various data structures and communication messages. The source code may be in a computer-executable form (e.g., via an interpreter), or the source code may be converted into a computer-executable form (e.g., via a translator, assembler, or compiler).

[0102] The computer-executable program code for performing operations of embodiments of the present disclosure may be written in an object-oriented, scripted, or non-scripted programming language, such as Java, Perl, Smalltalk, C++, or the like. However, the computer program code for performing operations of embodiments may also be written in conventional procedural programming languages, such as the "C" programming language or similar programming languages.

[0103] The computer program logic that implements all or part of the functionality previously described herein may execute at different times on a single processor (e.g., concurrently) or may execute at the same or different times on multiple processors, and may run under a single operating system process / thread or under different operating system processes / threads.

[0104] Consequently, the term “computer process” may generally refer to the execution of a set of computer program product instructions regardless of whether different computer processes are executing on the same or different processors, and regardless of whether different computer processes are running on the same operating system process / thread or different operating system processes / threads.

[0105] The computer program may be embodied in any form (e.g., source code form, computer-executable form, or intermediate form), either permanently or temporarily, in a tangible storage medium such as a semiconductor memory device (e.g., RAM, ROM, PROM, EEPROM, or flash-programmable RAM), a magnetic memory device (e.g., a floppy disk or hard disk), an optical memory device (e.g., a CD-ROM), a PC card (e.g., a PCMCIA card), or other memory device.

[0106] The computer program may be embodied in any form in a signal transmittable to a computer using any of various communication technologies, including, but not limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), networking technologies, and internetworking technologies.

[0107] The computer program may be distributed in any form as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink-wrapped software), preloaded with a computer system (e.g., on system ROM or hard disk), or distributed from a server or electronic notice board over the communications system (e.g., the Internet or World Wide Web).

[0108] Hardware logic (including programmable logic for use with a programmable logic device) that implements all or part of the functionality described hereinbefore may be designed using conventional manual methods or may be designed, captured, simulated, or documented electronically using various tools such as a computer-aided design (CAD), a hardware description language (e.g., VHDL or AHDL), or a PLD programming language (e.g., PALASM, ABEL, or CUPL).

[0109] Any suitable computer-readable medium may be used. The computer-readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device, or an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor medium.

[0110] More specific examples of the computer-readable medium include, but are not limited to, an electrical connection having one or more wires or other tangible storage medium such as a portable computer diskette, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), compact disk read-only memory (CD-ROM), or other optical or magnetic storage device.

[0111] The programmable logic may be embodied either permanently or temporarily in a tangible storage medium such as a semiconductor memory device (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), a magnetic memory device (e.g., a floppy disk or hard disk), an optical memory device (e.g., a CD-ROM), or other memory device.

[0112] The programmable logic may be embodied in a signal that is communicable to a computer using any of various communication technologies, including, but not limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), networking technologies, and internetworking technologies.

[0113] The programmable logic may be distributed as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink-wrapped software), preloaded with a computer system (e.g., on a system ROM or hard disk), or distributed from a server or electronic message board over the communications system (e.g., the Internet or World Wide Web).

[0114] Of course, some aspects may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments may be implemented as entirely hardware or entirely software.

[0115] Although certain exemplary aspects have been described and shown in the accompanying drawings, it is to be understood that such aspects are illustrative and that the examples are not limited to the specific constructions and arrangements shown and described, since various other changes, combinations, omissions, modifications and substitutions in addition to those set forth in the above paragraphs are possible.

[0116] Those skilled in the art will recognize that various adaptations, modifications, and / or combinations of the embodiments just described may be configured. Therefore, it should be understood that, within the scope of the appended claims, the disclosure may be practiced otherwise than as specifically described herein. For example, unless expressly stated otherwise, the steps of processes described herein may be performed in orders different from those described herein, and one or more steps may be combined, split, or performed concurrently. Those skilled in the art will also recognize, in light of this disclosure, that various examples or aspects described herein may be combined to form other examples. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2018 / 0370530 A1

[0004]

Claims

[1] Vehicle control device (10) for controlling a first vehicle (15) when a second vehicle (40) is detected at a location (2, 2a) of a predetermined road infrastructure (1) in the vicinity of the first vehicle (15), comprising: a state estimation unit (101) configured to estimate a state of the second vehicle (40); a vehicle interface unit (104) configured to to send a request to predict an expected trajectory of the second vehicle (40) together with the estimated state of the second vehicle (40) to a prediction device (20), and to receive an expected trajectory from the prediction device (20) which is predicted in view of information about the behavior of several vehicles at the location (2, 2a) of the second vehicle (40) and the estimated state of the second vehicle (40); and a vehicle control unit (103) configured to control a state of the first vehicle (15) based on the predicted expected trajectory received from the prediction device (20) and / or to display the predicted expected trajectory received from the prediction device (20) to a driver of the first vehicle (15). [2] The vehicle control device (10) according to claim 1, wherein the state estimation unit (101) is configured to use one or more signals from a plurality of sensors (100) mounted on the first vehicle (15) to estimate the state of the second vehicle (40), or receives the estimated state of the second vehicle (40) from an external state estimation unit. [3] The vehicle control device (10) according to claim 1 and / or 2, further comprising an environment estimation unit (102) configured to estimate information about an environment of the second vehicle (40) using one or more signals from a plurality of sensors (100) mounted on the first vehicle (15) and / or using information received from an external environment information unit. [4] The vehicle control device (10) according to at least one of claims 1 to 3, wherein the vehicle interface unit (104) is configured to send the estimated information about the surroundings of the second vehicle (40) to the prediction device (20) when the request to predict the expected trajectory of the second vehicle (40) is sent. [5] Vehicle control device (10) according to at least one of the preceding claims, wherein the vehicle interface unit (104) is configured to show information about the received predicted expected trajectory of the second vehicle (40) on a display (104a). [6] Vehicle (15) comprising a vehicle control device (10) according to at least one of the preceding claims and a plurality of sensors (100) mounted on the vehicle (15) and connected to the vehicle control device (10). [7] Prediction device (20) for predicting an expected trajectory of a second vehicle (40) detected at a location (2, 2a) of a given road infrastructure (1) in the vicinity of a first vehicle (15), comprising: a prediction interface unit (202) configured to receive a request for predicting an expected trajectory of the second vehicle (40) together with an estimated state of the second vehicle (40) from a vehicle control device (10) of the first vehicle (15) and to send a predicted expected trajectory of the second vehicle (40) to the vehicle control device (10); and a model unit (210) configured to predict the expected trajectory of the second vehicle (40) using a prediction model (211) in consideration of information about a behavior of a plurality of vehicles at the location (2, 2a) of the second vehicle (40) and the received estimated state of the second vehicle (40). [8] The prediction device (20) of claim 7, wherein the prediction interface unit (202) is connectable to one or more roadside units positioned at the location of the second vehicle that detect a trajectory and a condition of each of the plurality of vehicles thereat. [9] The prediction device (20) according to claim 8, wherein the model unit (210) is configured to determine the information about the behavior of the plurality of vehicles at the location (2, 2a) of the second vehicle (40) based on the trajectory and the state of each of the plurality of vehicles detected by the one or more roadside units (30a - 30e). [10] The prediction device (20) according to claim 9, wherein the model unit (210) is configured to determine a probability distribution of the detected trajectories at the location (2, 2a) of the second vehicle (40) as information about the behavior of the plurality of vehicles thereat. [11] The prediction device (20) according to claim 9 or 10, wherein the prediction model (211) is a machine learning / artificial intelligence (ML / AI) model, and the model unit (210) is configured to use the determined information about the behavior of the plurality of vehicles at the location (2, 2a) of the second vehicle (40) as training data for training the prediction model (211), and wherein the trained prediction model (211) is configured to predict the expected trajectory at the location (2, 2a) of the second vehicle (40). [12] The prediction device (20) of claim 11, wherein the model unit (210) comprises a training unit (212) configured to receive the training data and to label the received training data according to the state of each of the plurality of vehicles at the location (2, 2a) of the second vehicle (40). [13] Prediction device (20) according to at least one of claims 7 to 12, wherein the prediction interface unit (202) is configured to additionally receive information about an environment of the second vehicle (40), and the model unit (210) is configured to determine all potential trajectories of the second vehicle (40) on the basis of the received environment information for predicting the expected trajectory. [14] The prediction device (20) of claim 13, wherein the model unit (210) is configured to predict the expected trajectory by determining a probability for each potential trajectory of the second vehicle (40) and selecting the potential trajectory with the highest probability. [15] The prediction device according to at least one of claims 8 to 14, wherein the prediction interface unit (202) is configured to receive the state of the second vehicle (40) from at least one of the one or more roadside units (30a - 30e). [16] Prediction device according to at least one of claims 13 to 15, wherein the prediction interface unit (202) is configured to receive the additional information about the surroundings of the second vehicle from the vehicle control device (20) of the first vehicle (15) and / or from at least one of the one or more roadside units (30a - 30e). [17] System for controlling a first vehicle (15) when a second vehicle (40) is detected at a location (2, 2a) of a given road infrastructure (1) in the vicinity of the first vehicle (15), comprising a vehicle control device (10) according to at least one of claims 1 to 6; a prediction device (20) according to at least one of claims 7 to 16; and several roadside units (30a - 30e) positioned at several different locations (2, 2a) in the given road infrastructure. [18] The system of claim 17, wherein the plurality of roadside units (30a - 30e) form a roadside unit network (30) configured to exchange information between different roadside units (30a - 30e). [19] A method for controlling a first vehicle (15) when a second vehicle (40) is detected at a location (2, 2a) of a predetermined road infrastructure (1) in the vicinity of the first vehicle (15), comprising the steps of: - estimating a state of the second vehicle (40); - predicting an expected trajectory of the second vehicle (40) using a prediction model (211) in view of information about a behavior of several vehicles at the location (2, 2a) of the second vehicle (40) and the estimated state of the second vehicle (40); and - controlling a state of the first vehicle (15) based on the predicted expected trajectory of the second vehicle (40) and / or displaying the predicted expected trajectory to a driver of the first vehicle (15). [20] The method of claim 19, further comprising displaying control recommendations to a driver, such as including slowing down, continuing, emergency braking, and the like. [21] A computer program product storable in a working memory, comprising instructions which, when executed by a computer, cause the computer to perform the method according to claims 19 and / or 20.

Images