Simulation-based optimising of the use of teleoperators

Abstract

The invention relates to a system for determining an optimised assignment distribution of a plurality of teleoperators with a respective stationary control state (1) for a plurality of vehicles (3) requesting takeover by teleoperation, said system having a receiving unit (5) designed to receive requests for takeover and to receive real-time traffic data in a traffic area around a respective one of the requesting vehicles (3), and having a computing unit (7) which is connected to the receiving unit (5) and which is designed, by means of the information obtained from the receiving unit (5) and a digital map of the respective traffic area, to determine, by means of simulation and / or by carrying out a pre-trained machine learning model, a measure for the impedance of the traffic flow depending on a time-related non-takeover of the respective vehicle (3) by teleoperation and to assign the teleoperator to the requesting vehicles (3) by minimising a cost function based on the measures for the impedance of the traffic flow.

Description

[0001] SIMULATION-BASED OPTIMIZATION OF THE USE OF TELEOPERATING VEHICLE DRIVERS

[0002] The invention relates to a system for determining an optimized allocation distribution of a plurality of teleoperating vehicle drivers with a respective stationary control station to a plurality of vehicles requesting takeover by teleoperation within the same period and previously operated in an automated manner.

[0003] Typical conventional motor vehicles such as passenger cars are designed to be driven by a person inside the vehicle. With increasing levels of automation, the role of this person changes from that of a driver to that of active vehicle control, increasingly to the sole task of monitoring the driving maneuvers and control interventions performed independently by the vehicle. At the forefront of this development are fully automated vehicles, which can autonomously perform not only individual maneuvers but also drive an entire planned route. While a human driver in a non-automated vehicle is the vehicle's sole decision-making body, they also assume at least some of the role of actuators, with their movements mechanically specifying control variables (at most with direct assistance).However, with the increasing degree of vehicle automation, actuators must be provided, for example electrical or hydraulic actuators, which are all controlled by control electronics so that a vehicle computer can transmit corresponding commands to the actuators, which then implement them mechanically. This circumstance allows an interface on a digital control unit or directly on the vehicle's actuators to be controlled from an external control center. This opens up the possibility of teleoperation of the automated vehicle, in which a human driver of the vehicle does not have to take a seat in the vehicle itself, but can sit outside the vehicle at a stationary workstation and issue commands to this vehicle interface, which are then transmitted to the vehicle for execution via data transmission, especially wirelessly.In addition, relevant vehicle information (e.g., a video stream with images from the driver's seat) can be transmitted to the remotely controlling driver. If the teleoperation takes place in an environment with other road users, the human driver can initiate necessary vehicle movements (through lateral / longitudinal control) to avoid traffic obstructions and dangerous situations while reaching a destination by detecting the vehicle's local surroundings and the other road users within them.

[0004] In this context, DE 10 2021 123 234 A1 relates to a teleoperating driver's workstation for a teleoperated motor vehicle, wherein the motor vehicle has a front camera, a rear camera, a left side camera, which is optionally directed towards a rear left side of the motor vehicle, and a right side camera, which is optionally directed towards a rear right side of the motor vehicle, and wherein the motor vehicle is designed to send images recorded by the front camera, the rear camera, and the left and right side cameras to the teleoperating driver's workstation, wherein the teleoperating driver's workstation is designed to change a display of the images received from the motor vehicle depending on an orientation of a head of a teleoperating driver and / or depending on a viewing direction of the teleoperating driver.

[0005] If fundamentally automated vehicles have difficulty making decisions or are unable to handle a particular traffic situation, drivers from stationary test benches can be called upon to temporarily take over control of the vehicle remotely. There are several reasons why the use of a remotely controlled driver can be useful for fundamentally automated vehicles: If a complex situation exists that overwhelms the automated vehicle's algorithm for independent driving control, a human driver can safely guide the vehicle through the situation from a stationary control station by manually taking over the vehicle and remotely controlling it accordingly. In other cases, the vehicle's driving control system may not be able to make an optimal decision itself.Here, too, the driver can assist from the stationary control station by making appropriate decisions based on human experience and intuition. This can also save costs, as an automated vehicle can in principle operate autonomously, even if it cannot yet be guaranteed to handle every traffic situation autonomously. Furthermore, human experience can be more valuable than machine experience for some situations, for example, in being able to respond appropriately to other human interactions in the vicinity of the remote-controlled vehicle. For example, if there are hand signals from people standing around or verbal instructions from passersby, the vehicle may not be able to interpret them, but the driver from a distance can.Even in the event of technical problems, a human driver may still be able to move the vehicle safely or relocate it to a safe location, while the algorithm of the automated driving control system can no longer do so. Overall, the use of teleoperating drivers for automated vehicles can, in certain situations, increase safety, boost efficiency, and reduce costs.

[0006] If an automated vehicle encounters a situation where a teleoperating driver is required to intervene but none is available, the consequences may vary depending on the situation and environment:

[0007] In a construction site or similar situations where the automated vehicle may come into contact with other vehicles or pedestrians, this can lead to dangerous situations. Without human control, there is a possibility that the vehicle could endanger or damage other vehicles or pedestrians. In such a situation, the autonomous vehicle may not be able to find a solution independently and may have to remain in place until a teleoperating driver is available. In some cases, this could lead to traffic disruptions or even traffic chaos. Furthermore, the absence of a teleoperating driver could also lead to delays or even the failure of services provided by automated vehicles.For example, an automated delivery van may be unable to deliver goods to customers if the vehicle encounters a situation where a teleoperating driver is required but none is available. Overall, this demonstrates the importance of having enough teleoperating drivers available to ensure that automated vehicles can operate safely and effectively in all situations.

[0008] If more teleoperating drivers are required than are available, this could lead to a potential bottleneck or shortage in the use of automated vehicles. This means that in situations where a teleoperating driver is required to control one of the automated vehicles, there may not be a suitable person available to fill that role. This, in turn, could prevent the automated vehicle from providing the desired route or service, resulting in delays or even service failure. Furthermore, the delay in providing teleoperating drivers could result in a person having to wait an extended period for assistance in an emergency situation.To solve this problem conventionally, more teleoperating vehicle drivers would have to be trained and hired to ensure that sufficient personnel are available to meet the required reliability of a fundamentally autonomous vehicle fleet.

[0009] The object of the invention is to solve this problem technically, in particular to improve traffic flow around highly automated vehicles requiring teleoperated takeover in order to avoid traffic disruptions.

[0010] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0011] A first aspect of the invention relates to a system for determining an optimized allocation distribution of a plurality of teleoperating vehicle drivers with a respective stationary control station to a plurality of vehicles requesting takeover by teleoperation within the same period and previously operated in an automated manner, comprising a receiver unit designed to receive requests from vehicles for takeover by teleoperation and to receive real-time traffic data in a traffic area around a respective one of the requesting vehicles, and comprising a computing unit connected to the receiver unit, which is designed toUsing the information received from the receiver unit and a digital map of the respective traffic area, by means of simulation and / or by executing a pre-trained machine learning model, to determine a measure of the traffic flow obstruction depending on a time-related non-takeover of the respective vehicle by teleoperation, and to assign the drivers to the requesting vehicles by minimizing a cost function based on the measures of the traffic flow obstruction.

[0012] The assumption is made that within one and the same period of time, a large number of vehicles request takeover via teleoperation, whereby the number of drivers available for teleoperation is fewer than the number of requesting vehicles. If a new request for takeover is made by another previously automated vehicle to the processing unit, this is entered into the system as a new order so that it is processed according to its urgency compared to the other orders and the availability of the teleoperators. The time-related non-takeover does not mean that a takeover will never take place; rather, it means that an appropriate consequence is determined depending on the period of non-takeover. For example, the respective corresponding measure for non-takeover is determined in second increments.The cost function increases in correlation with this as the measure also increases with time.

[0013] The tasks to be executed can vary greatly in their criticality. However, at least an estimate is made of the consequences of not taking over the respective vehicle by teleoperation for certain periods. For this estimate, a traffic simulation and / or a pre-trained machine learning model such as an artificial neural network is executed. The desired measure of the disruption to traffic flow in the local traffic space around a respective affected vehicle can be output as a single value or by statistical variables for the affected vehicles, for example, described by mean and variance or, equivalently, a standard deviation. To carry out such an estimate and determine such a measure, correspondingly powerful computing units are necessary, especially when using a traffic simulation.If a pre-trained machine learning model is used, it is important to ensure that it has been trained with a sufficiently broad set of training data to ideally handle all real-world situations. The machine learning model can also be updated during its product lifetime and trained with new data to continuously improve the accuracy of the estimates. The extent of the congestion is determined either as a constant value (number of waiting cars per unit of time) or as a function of time (number of waiting cars per unit of time), since an increasing traffic jam can lead to further congestion.

[0014] If the computing unit uses a simulation, the environment of the requesting vehicle is modeled using a digital map in a suitable traffic simulation that is as accurate as possible and, if necessary, expanded to include static obstacles such as construction sites, etc. A virtual vehicle used in the simulation, which is to be secured by a teleoperating driver, is aligned analogously to reality and positioned in the simulation environment.

[0015] Multi-objective optimization can also be applied to minimize the cost function. In this case, in addition to the degree of disruption to traffic flow, the criticality of the respective vehicle's position can also form a term in the cost function based on properties of the position itself. Furthermore, the duration of the intervention until the entire problem is solved, as well as the time required to reach a safe position that does not bind other road users, can be taken into account in the cost function with a corresponding term.If the simulation determines how many requesting vehicles have to wait per unit of time, the current local traffic flow is determined from current data. To determine the expected future local traffic flow, the surrounding traffic is simulated and determined, determining how long other road users in the traffic area under consideration need to travel the local routes around the respective requesting vehicle. This waiting time is compared to the time required without the traffic obstruction caused by the respective requesting vehicle. Either data is available on how long the road users needed before the obstruction caused by the respective requesting vehicle, or an additional reference simulation must be performed without the obstructing vehicle.Furthermore, the criticality of the current position of a requesting vehicle for the takeover is preferably taken into account in the respective simulation, as well as how other road users react when they encounter the obstructing vehicle to determine the criticality. This simulation is preferably carried out multiple times with varying surrounding traffic parameters. Suitable metrics from this field, such as time-to-collision, are used to determine the criticality. Criticality will be high, for example, if the vehicle has broken down behind an obscured area.

[0016] Through parallelization and multiple execution, various reactions of other road users can be analyzed in the simulation in order to obtain a statistical distribution and / or a worst-case estimate for the respective measures.

[0017] To determine the further term of the cost function with reference to the measure of the time required to steer the vehicle to its destination, the requesting vehicle depicted in the simulation is steered to its destination in the simulation. A large proportion of this time can be due to turning in a tight space, for example. Since the autonomous control system of the vehicle to be controlled was apparently unable to carry out this driving operation, it may be that more information needs to be made available to the virtual vehicle in the simulation than the vehicle's sensors can detect, or that more leeway needs to be offered with regard to compliance with traffic regulations or that the permissible criticality level needs to be increased. This can ensure that the destination is reached and that a finite time can be determined until the requesting vehicle reaches the destination.

[0018] Determining the time required to reach a safe position for parking the vehicle can be done using the same principle, except that the safe position must first be determined as the temporary destination. To do this, a location is preferably sought that is close to the current location, but takes up as little space as possible in a lane, and if it is present, is clearly visible. The simulation makes it possible to run through various scenarios and record the required times. The simulations do not have to run in real time, but can also run faster with sufficient computing power.

[0019] If, on the other hand, the machine learning model is used, speed advantages can be used to estimate the measure of traffic flow disruption. Optionally, other measures included in the cost function and all available values, such as those used in the simulation, can be used. Based on the relevant traffic space surrounding a particular vehicle request, a preliminary estimate must be made as to whether the machine learning model has been trained for similar traffic sections and whether the results are therefore trustworthy. If this is the case, the machine learning model can output the corresponding values ​​for the measures. If this is not the case, either placeholder values ​​must be assumed or a simulation must be run. The simulation method can generally also be used to train the machine learning model.

[0020] By separately considering the waiting times until a remotely controlled driver takes over at a stationary control station for teleoperation of the vehicles, traffic flow is disrupted as little as possible. The method thus solves the problem that the number of requested remotely controlled drivers exceeding the number of available remotely controlled drivers can result in longer traffic delays than are necessary through a clever distribution of drivers among the vehicles requiring teleoperation. Waiting times and other disruptions to traffic flow, especially for other road users, are therefore minimized as much as possible.

[0021] According to an advantageous embodiment, the computing unit is designed to determine a measure of a time period for solving the problem relating to the respective vehicle in order to determine a further term of the cost function.

[0022] According to a further advantageous embodiment, the computing unit is designed to determine a measure of a time period which is necessary to transfer the respective vehicle into a position which does not disrupt the traffic flow in order to determine a further term of the cost function.

[0023] According to a further advantageous embodiment, the computing unit is designed to determine a measure of a fundamental criticality of the current position in which the respective requesting vehicle is located before being taken over by teleoperation in order to determine a further term of the cost function.

[0024] According to a further advantageous embodiment, statistical quantities are obtained as a result of the simulation, in particular expected value and standard deviation.

[0025] According to a further advantageous embodiment, the receiver unit is designed to receive sensor data of a respective vehicle and to transmit it to the computing unit, wherein the computing unit is designed to merge the real-time traffic data with the sensor data of a respective vehicle.

[0026] According to a further advantageous embodiment, the computing unit is designed to generate the cost function by modeling a dynamic queuing problem.

[0027] To determine the optimal sequence using numerical methods, the problem is interpreted as a dynamic queueing problem. In contrast to static queueing problems, the problem can change dynamically, e.g., due to the arrival of new tasks, the loss of agents (here, vehicle drivers), or changes in priorities or adjustments to processing times. Detailed descriptions can be found, for example, in the publication: Runzheimer, B. (1999). Queueing Theory — Basic Concepts and Possible Applications. In: Operations Research. Modern Business Books, vol. 7. Gabler Verlag, Wiesbaden. https: / / doi.org / 10.1007 / 978-3-322-94495-5_5.

[0028] According to a further advantageous embodiment, the computing unit is designed to consider restrictions when minimizing the cost function. According to a further advantageous embodiment, the computing unit is designed to implement the takeover of vehicles by teleoperation as a restriction with a measure of the disruption to traffic flow caused by non-takeover above a predetermined threshold without a waiting time for the respective affected vehicle.

[0029] According to a further advantageous embodiment, the computing unit is designed to comply with the restriction of undercutting a predetermined maximum waiting time for the vehicles requesting the takeover.

[0030] Further advantages, features, and details will become apparent from the following description, which – where appropriate with reference to the drawings – describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0031] It shows:

[0032] Fig. 1 : A system according to an embodiment of the invention.

[0033] Fig. 1 shows a system for determining an optimized allocation distribution of a plurality of teleoperating vehicle drivers with a respective stationary control station 1 to a plurality of previously automated vehicles 3 requesting takeover by teleoperation within the same period.

[0034] In the exemplary traffic area, a large number of automated vehicles 3 (only one is shown for the sake of simplicity) are traveling in autonomous mode, i.e., without a human driver on board. A subset of these automated vehicles 3, particularly in the case of a large number of automated vehicles 3 operating simultaneously, is expected to transmit a request for manual remote takeover by a human driver to the system via its receiver unit 5. Based on this expectation, several vehicle drivers are already ready, each of which has its own stationary control station 1 (only one is shown for the sake of simplicity), which can send commands to a taken over and remotely operated vehicle 3 by transmitting radio signals.In return, the respective stationary control station 1 receives environmental data, such as camera images, from the vehicle 3 it is controlling, so that the human driver can act and react accordingly to the current situation around the vehicle 3. For this purpose, a requesting vehicle 3 transmits its coordinates, its orientation, sensor data such as camera data, lidar data, radar data, information about its own vehicle 3, in particular a vehicle model, and the current destination, via the receiver unit 5 to the system's computing unit 7. In the computing unit 7, the received information can be assigned to a digital map and combined with other information, such as weather data, traffic monitoring data, real-time traffic data, and, if available, data from other vehicles located in the relevant area around the requesting vehicle 3.It can now be assumed that more requests 3 are received in the system in a given period of time than the number of available vehicle drivers in a given control station 1 allows to answer. The solution is based on estimating the consequences of a teleoperator's non-intervention. This estimation serves as the basis for an optimization algorithm to calculate and specify an optimal approach for the available teleoperators.For each of the requesting vehicles 3 for which no driver has yet taken control in their stationary control station 1, the computing unit 7 carries out a simulation using the information received from the receiving unit 5 and a digital map of the respective traffic area in order to determine, for each of the requesting vehicles 3, a respective measure of the obstruction of the traffic flow by the other road users in this respective locally limited traffic area, depending on a non-takeover of the respective vehicle 3 by teleoperation for various time periods. The allocation of the vehicle drivers to the requesting vehicles 3 is then carried out by the computing unit 7 by globally minimizing a cost function, which is a sum of summands based on the respective measures of the obstruction of the respective local traffic flow.

[0035] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that exemplary embodiments are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

[0036] List of reference symbols

[0037] 1 control station

[0038] 3 vehicles

[0039] 5 Receiver unit

[0040] 7 Computing unit

Claims

Patent claims 1. System for determining an optimized allocation distribution of a plurality of teleoperating vehicle drivers with a respective stationary control station (1) to a plurality of vehicles (3) requesting takeover by teleoperation within a period of time and previously operated automatically, comprising a receiver unit (5) designed to receive requests from vehicles (3) for takeover by teleoperation and to receive real-time traffic data in a traffic area around a respective one of the requesting vehicles (3), and comprising a computing unit (7) connected to the receiver unit (5), which is designed toto determine a measure of the obstruction of the traffic flow depending on a time-related non-takeover of the respective vehicle (3) by means of teleoperation using the information received from the receiver unit (5) and a digital map of the respective traffic area by means of simulation and / or by executing a pre-trained machine learning model, and to assign the vehicle drivers to the requesting vehicles (3) by minimizing a cost function based on the measures of the obstruction of the traffic flow.

2. System according to claim 1, wherein the computing unit (7) is designed to determine a measure of a time period for solving the problem relating to the respective vehicle (3) in order to determine a further term of the cost function.

3. System according to one of the preceding claims, wherein the computing unit (7) is designed to determine a measure of a time period which is necessary to transfer the respective vehicle (3) into a position which does not disrupt the traffic flow in order to determine a further term of the cost function.

4. System according to one of the preceding claims, wherein the computing unit (7) is designed to determine a measure of a fundamental criticality of the current position in which the respective requesting vehicle (3) is located before takeover by teleoperation in order to determine a further term of the cost function.

5. System according to one of the preceding claims, wherein statistical quantities are obtained as a result of the simulation, in particular expected value and standard deviation.

6. System according to one of the preceding claims, wherein the receiver unit is designed to receive sensor data of a respective vehicle (3) and to transmit it to the computing unit (7), wherein the computing unit (7) is designed to merge the real-time traffic data with the sensor data of a respective vehicle (3).

7. System according to one of the preceding claims, wherein the computing unit (7) is designed to generate the cost function by modeling a dynamic queuing problem.

8. System according to one of the preceding claims, wherein the computing unit (7) is designed to take restrictions into account when minimizing the cost function.

9. System according to claim 8, wherein the computing unit (7) is designed to carry out the takeover of vehicles (3) by teleoperation as a restriction with a measure of the obstruction of the traffic flow by non-takeover above a predetermined limit value without waiting time for the respective affected vehicle (3).

10. System according to one of claims 8 to 9, wherein the computing unit (7) is designed to comply with the restriction of falling below a predetermined maximum waiting time for the vehicles (3) requesting the takeover.

Images