Computer-implemented method and computer program for controlling the longitudinal or lateral guidance of a driving system and hardware unit for a driving system

Abstract

Computer-implemented method for controlling the longitudinal or lateral guidance of a vehicle system, wherein a model predictive control algorithm (MPC) is used at first sampling times (t k MPC , t k+1 MPC ) is executed and each future trajectory (x k+1 MPC ) of the driving system via a sampling time at the first sampling points (t k MPC , t k+1 MPC ) subdivided time horizon (T) and control signals (u MPC ) to realize this trajectory (V1); by determining at each of the first sampling times (t k MPC ) a current state of the driving system is determined based on data acquired by sensors of the driving system (V2); by minimizing a predefined cost function over the time horizon (T) a sequence of control signals (u MPC) to drive along the trajectory (V3) by solving a current-state-dependent optimization problem (V4); which consists of the sequence of control signals (u MPC ) received first control signal (u1) MPC ) at least one actuator (Act) of the driving system is provided for longitudinal or lateral guidance (V5); the time horizon (T) on the following of the first sampling times (t k+1 MPC ) is shifted (V6); the preceding steps are repeated, characterized in that the time intervals (Δt) between successive first sampling times (t k MPC , t k+1 MPC ) each into second sampling times (t̂ j AI , t̂ i AI , t̂ l AI ) are subdivided; a machine learning model (AI) that was trained on data concerning the states of the driving system in traffic scenarios, control signals (û AI) to predict the trajectory control of the driving system, used and in the second sampling times (t̂ j AI , t̂ i AI , t̂ l AI ) is executed and each second control signals (û2) AI ) determined (V7); the second control signals (û2 AI ) are provided to the actuator (Act) and executed by it (V8); at the first sampling times (t k MPC , t k+1 MPC ) the first control signals (u1 MPC ) with the respective parallel second control signals (û2 AI ) are compared (V9) and if the second control signals (û2 AI ) from the first control signals (u1 MPC ) deviate, the first control signals (u1 MPC ) are executed by the actuator (Act) (V10).

Description

[0001] The invention relates to a computer-implemented method and a computer program for controlling the longitudinal or lateral guidance of a driving system. Furthermore, the invention relates to a hardware unit for a driving system.

[0002] The following definitions, descriptions and statements retain their respective meaning for and apply to the entire disclosed subject matter of the invention.

[0003] Model predictive control, also called MPC, is a well-known algorithm for controlling systems. It is known that such an algorithm runs slowly. Furthermore, it is difficult to run such an algorithm on an embedded system due to limitations in the embedded system's computing and memory capacity. It is known that MPC is a reliable method, and its stability can be demonstrated. The disadvantage is that MPC is very computationally intensive and requires a relatively long runtime.

[0004] Instead of a model-predictive control algorithm, an artificial intelligence system (AI), such as a machine learning model, can also be used for control. See, for example, Dimitri Bertsekas, Reinforcement Learning and Optimal Control, Athena Scientific, 2019, ISBN 1886529396, 9781886529397. The machine learning model is, for example, an artificial neural network. The machine learning model is fast compared to the model-predictive control algorithm, but there is currently no evidence of the stability of artificial intelligence algorithms for controlling systems.

[0005] There are known approaches in which an already developed model-predictive control algorithm is to be replaced with an algorithm from the field of artificial intelligence, see, for example, Murad Dawood et al., Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Control, arXiv:2210.01525v1. It is also known to have the model used in the model-predictive control algorithm learned by an artificial neural network, see Tim Salzmann et al., Real-time Neural MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms, arXiv:2203.07747v2.

[0006] Further prior art is disclosed, for example, in DE 10 2019 214 923 A1, DE 10 2019 214 935 A1, DE 10 2021 205 037 A1 and EP 3 498 555 A1.

[0007] The invention is based on a generic model-predictive control algorithm, as disclosed, for example, in https: / / www.ist.uni-stuttgart.de / research / group-of-frank-allgoewer / model-predictive-control / or https: / / www.mathworks.com / help / mpc / gs / what-is-mpc.html. The underlying problem of the invention is how control algorithms, for example in safety-critical systems, can be calculated stably and quickly. Model predictive control is stable, but not fast. A trained machine learning model is fast, but not stable.

[0008] The subject matter of the independent and subordinate claims each solves this problem. Advantageous embodiments of the invention emerge from the definitions, the subclaims, the drawings, and the description of preferred embodiments.

[0009] According to one aspect, the invention provides a computer-implemented method for controlling the longitudinal or lateral guidance of a driving system. Longitudinal guidance includes, for example, positive acceleration and braking. Lateral guidance includes, for example, steering. The driving system can relate to individual components of a driving system. Components of the driving system are, for example, the drive, chassis, or sensor technology including environment detection sensors. Environment detection sensors are, for example, cameras, radar, lidar, or acoustic sensors. The sensor technology can also comprise at least one inertial measuring unit. The driving system can also relate to a vehicle as such. The vehicle is, for example, a passenger vehicle or a commercial vehicle. For example, the vehicle is a shuttle for transporting people or goods. According to one aspect, the vehicle can be operated automatically. For example, the vehicle is an autonomous passenger shuttle.

[0010] In the method, a model-predictive control algorithm is executed at first sampling times. At each of the first sampling times, a future trajectory of the driving system is determined over a time horizon divided into the first sampling times, along with control signals for implementing this trajectory. According to one aspect, the first sampling times are equidistant. For example, a time horizon of 60 s can be predicted, which is divided into sampling times or time steps of 1 s each. The control signals can be logic levels, for example, electrical voltage levels.

[0011] The model predictive control algorithm can comprise linear or nonlinear model predictive control. In model predictive control, a discrete-time dynamic model of the process to be controlled is used to calculate the future behavior of the process depending on the input data.

[0012] The basic idea of ​​the model predictive control algorithm is to predict the future behavior of the controlled system over a finite time horizon and to calculate an optimal control input that minimizes an a priori defined cost functional while simultaneously satisfying the given system constraints. More precisely, the control input is calculated by solving a finite-horizon optimal control problem at each sampling time. The first part of the resulting optimal input trajectory is then applied to the system until the next sampling time, shifting the time horizon and repeating the entire procedure cyclically.

[0013] At each of the first sampling times, a current state of the driving system is determined based on data acquired by sensors of the driving system. The state can include the orientation, pose and / or movement state, for example the magnitude and direction of the driving speed, of the driving system relative to objects in the driving system's environment, such as vehicles, pedestrians, vegetation objects, infrastructure objects. The state can be characterized by speed, acceleration, roll, pitch and / or yaw angles, steering angle, component temperatures, engine, transmission and / or energy state data and / or target torques. The state can relate to an environment model. The data can be acquired using environment detection sensors of the driving system, for example cameras, radar, lidar and / or acoustic sensors.

[0014] By minimizing a given cost function over the time horizon, a sequence of control signals for driving along the trajectory is determined by solving an optimization problem dependent on the current state. The model-predictive control algorithm can include a model of the state and driving task of the driving system, which is optimized using the cost function.

[0015] The first control signal obtained from the sequence of control signals is provided to at least one actuator of the driving system for longitudinal or lateral guidance. The actuator converts the control signals into effects. The actuator can be, for example, an electric motor, a pneumatic cylinder, a hydraulic cylinder, or a valve. The actuator acts, for example, on the drive train, brakes, shock absorbers, or suspension. For example, a perception functional unit of the driving system comprising environment detection sensors can plan a future trajectory. The trajectory is implemented by the actuators.

[0016] The time horizon is then shifted to the next of the first sampling times, and the previous steps are repeated, specifically cyclically. For example, to predict a time horizon of 60 s, the next 60 s are calculated in advance in each time step of, say, 1 s. This ensures security, robustness, and guarantees.

[0017] The method is characterized in that the time intervals between consecutive first sampling times are each divided into second sampling times. According to one aspect, the second sampling times are equidistant. For example, the time interval between consecutive first sampling times is 1 s long, and the second sampling times are each spaced 10 ms apart.

[0018] A machine learning model trained on data concerning driving system states in traffic scenarios to predict control signals for trajectory control of the driving system is used and executed at the second sampling time points. Compared to the model-predictive control algorithm, the trained machine learning model has a shorter runtime and is less computationally intensive. The machine learning model determines second control signals in each case. Machine learning is a technology that teaches computers and other data processing devices how to perform tasks by learning from data, rather than being programmed for the tasks. The machine learning model is, for example, an artificial neural network.

[0019] The second control signals are provided to the actuator and executed by it. For example, the machine learning model is executed 100 times in one computing cycle of the model-predictive control algorithm, which lasts, for example, 1 s. In this example, using the machine learning model, 100 second control signals are obtained within 1 s, each 10 ms apart, with the model-predictive control algorithm only being executed every second. Without the machine learning model, the model-predictive control algorithm would have to be executed every 10 ms to determine 100 control signals within 1 s, which would be very computationally intensive. With the relatively low sampling rate for the machine learning model disclosed here, the machine learning model can be executed frequently. This allows fast decisions regarding trajectory control to be made on the hardware unit disclosed here.By combining this with the model-predictive control algorithm, which runs at a relatively slow rate, stability is achieved.

[0020] At the first sampling times, the first control signals are compared with the respective temporally parallel second control signals. If the second control signals deviate from the first control signals, the first control signals are executed by the actuator. This offers the advantage that the model-predictive control algorithm can intervene if the machine learning model calculates results that do not correspond to stability and could therefore violate safety requirements. When comparing the first control signals with the second control signals, according to one aspect, a tolerance range for the first control signals can be specified, within which the second control signals may lie during the comparison and are executed by the actuator.

[0021] According to one aspect, the interval between two consecutive first sampling times corresponds to the time the driving system needs before it transitions into an unsafe state. From the safety analysis of the driving system, it is known that there is a time the driving system needs before it transitions into an unsafe state. This time can then be used as the runtime for the model-predictive control algorithm. To ensure that the driving system continues to be comfortable or efficient to drive, the machine learning model takes on the role of the high-frequency algorithm in order to achieve the most comfortable and efficient result possible through fast control. The model-predictive control algorithm assumes the task of safe control, which can run significantly slower than the machine learning model.

[0022] According to another aspect, the sequence of the second sampling times is high-frequency relative to the sequence of the first sampling times. For example, the frequency at which the machine learning model is executed is 100 times higher than that of the model-predictive control algorithm.

[0023] Another aspect involves regulating the longitudinal and lateral guidance of the driving system. This enables controlled, highly automated operation of the driving system.

[0024] According to a further aspect, the invention provides a computer program for controlling the longitudinal or lateral guidance of a driving system. The computer program comprises program instructions that cause the computer to execute the method disclosed herein when the computer program is loaded onto the computer or executed by the computer. The computer program can be provided as interpretable code that is loaded onto the computer, for example, directly into an internal memory of the computer, or as compiled code that is executed by the computer. During interpretation, an interpreter executes operations for the program to be interpreted during runtime. The interpreter is, for example, another computer program that can enforce compliance with higher-level safety guidelines during the runtime of the computer program disclosed herein.Compilation is the translation of a program written in one source language into a program in another language that is equivalent in many respects. Provision can be made, for example, on a machine-readable medium on which the computer program is stored, such as a non-volatile storage medium, or via a data carrier signal. The program instructions can be machine instructions, source code, or object code, written in assembly language, an object-oriented programming language, such as C++, or a procedural programming language, such as C.

[0025] According to a further aspect, the invention provides a hardware unit for a driving system. The hardware unit is also called a high-performance unit, abbreviated to HPU. The hardware unit comprises at least one central processor, a memory, at least one hardware accelerator for machine learning models, and a data transmission device via which the central processor, the memory, and the hardware accelerator for data transmission are connected. The microarchitecture of the hardware unit is specialized for executing the model-predictive control algorithm and the machine learning model disclosed here. According to one aspect, the hardware unit comprises several central processors, also called central processing units, abbreviated to CPU. The memory can, for example, comprise a flash EEPROM (electrically erasable programmable read-only memory) memory unit and / or a RAM (random access memory) memory unit.Hardware accelerators for machine learning models include, for example, low-precision arithmetic, for example, special data formats including half-precision format and bfloat16 floating-point format, and / or special microarchitectures for parallelized processing of calculations or sequences. Special microarchitectures for parallelized processing can be microarchitectures of graphics processors, also called graphical processing units (GPUs), or architectures based on them. The hardware accelerator improves a real-time application of the machine learning model. The data transmission device can be a system bus, for example, a high-speed bus.

[0026] The hardware unit is, for example, an embedded system, such as an electronic control unit. The hardware unit can be implemented as a single-chip system, also called a system-on-chip (SoC). In a single-chip system, all or at least a large portion of the functions of a programmable electronic system are integrated on one chip, i.e., a die.

[0027] The computer program disclosed here is loadable into memory.

[0028] By means of the data transmission device, when the computer program is loaded, the program instructions of the computer program relating to the model predictive control algorithm disclosed here are loaded onto the central processor or executed by it.

[0029] By means of the data transmission device, when the computer program is loaded, the program instructions of the computer program relating to the machine learning model disclosed here are loaded onto the hardware accelerator or executed by it.

[0030] This relieves the central processor of computational load, freeing up computing capacity for the model-predictive control algorithm. The hardware accelerator allows the machine learning model to run quickly and energy-efficiently.

[0031] The hardware unit disclosed here, in particular by executing the model-predictive control algorithm on the central processor and executing the machine learning model on the hardware accelerator, provides safety functions. According to one aspect, the hardware unit is designed to be operationally safe to ASIL-D level, the highest automotive safety integrity level. Thus, the hardware unit is configured for data-intensive automotive applications, such as domain and zone control.

[0032] The classic model-predictive control algorithm serves as a safety or monitoring path and ensures the required system boundaries or modulation limits. Executing the machine learning model on the hardware accelerator enables relatively fast iteration loops compared to the model-predictive control algorithm and ensures system performance. Using the data transmission device, the machine learning model is cyclically corrected using the results of the model-predictive control algorithm. This advantageously prevents numerical or algorithmic divergences.

[0033] The hardware unit disclosed here enables better performance, higher security, energy efficiency and resource conservation.

[0034] According to another aspect, the hardware unit comprises several of the hardware accelerators. The machine learning model is loaded or executed in a distributed manner across the hardware accelerators. This optimizes computing resources and the runtime of the machine learning model.

[0035] A further aspect of the invention relates to the use of the computer-implemented method, the computer program or the hardware unit disclosed here in an automated operation of a driving system.

[0036] The invention is illustrated in the following exemplary embodiments. They show: Fig. 1 an embodiment of an MPC prediction for a trajectory and real system, Fig. 2 an embodiment of an MPC prediction for a control signal and real system, Fig. 3 an embodiment of a control system disclosed here with MPC and AI prediction, Fig. 4 an embodiment of a hardware unit disclosed here and Fig. 5 a schematic representation of a method disclosed here.

[0037] In the figures, identical reference numerals designate identical or functionally similar parts. For clarity, only the reference parts relevant to the respective understanding are marked in the individual figures.

[0038] Fig. 1 shows a trajectory x(t) with real points x k plotted over a time t. A model predictive control algorithm MPC determines over a time horizon T, which is divided into first sampling times t k MPC which are spaced apart by a time interval Δt, each point x k+1 MPC the trajectory x(t), see also Fig. 3. The time horizon T is shifted by the time interval Δt in the positive time direction after each calculation cycle of the model predictive control algorithm MPC.

[0039] Fig. Figure 2 shows a control signal u(t) plotted against time t. The model predictive control algorithm MPC determines a sequence of control signals u in each time horizon T MPC each with a first control signal u1 MPC .

[0040] Fig. 3 shows a division of the time interval Δt = t k+1 MPC - t k MPC into high-frequency second sampling times t̂ i AI , in each of which a trained machine learning model AI is executed and the sequence of control signals û AI determined with a second control signal û2 AI . The control signal û AI of the machine learning model result in points x̂ i AI the trajectory x(t). As long as the points x̂i AI of the machine learning model AI within a given tolerance range Δ relative to the points x determined by the model predictive control algorithm MPC k MPC the model predictive control algorithm MPC does not intervene. If the first sampling time t k+t MPC the point x̂ determined with the machine learning model AI k+1 AI outside the tolerance range Δ of the point x determined by the model predictive control algorithm MPC k+1 MPC , the underlying control signal determined with the machine learning model AI û k+1 AI , but the control signal u determined with the model predictive control algorithm MPC k+1 MPC executed by an actuator act for vehicle control.

[0041] Fig. Figure 4 shows an embodiment of a hardware unit HPU. The hardware unit HPU comprises, for example, two central processors CPU1 and CPU2. The model predictive control algorithm MPC is executed on the central processor CPU1, comprising determining the points x k MPC the trajectory x(t) and the first control signals u1 MPC .

[0042] The hardware unit HPU also includes, for example, two hardware accelerators AI Acc1 and AI Acc2 for the machine learning model AI. The machine learning model AI is executed distributed across the hardware accelerators Acc1 and AI Acc2. The points x̂ i AI and the second control signals û2 AI certainly.

[0043] The hardware unit HPU further comprises a memory Mem in which the computer program disclosed here, comprising program instructions for the model predictive control algorithm MPC and program instructions for the machine learning model AI, is stored.

[0044] The hardware unit HPU further comprises a data transmission device X-Bar, by means of which the central processors CPU1, CPU2, the memory Mem and the hardware accelerators are connected for data transmission.

[0045] The first control signals u1 MPC and the second control signals û2 AI are compared in a comparison unit Com. If the second control signals û2 AI outside a tolerance range Δ of the first control signals u1 MPC the actuator is controlled with the first control signals u1 MPC controlled.

[0046] Fig.Figure 4 shows the method disclosed here. In a method step V1, the model predictive control algorithm MPC is executed at first sampling times t k MPC , t k+1 MPC is executed and a future trajectory x k+1 MPC of the driving system over the first sampling times t k MPC , t k+1 MPC divided time horizon T and control signals u MPC intended to realize this trajectory.

[0047] In a process step V2, at each of the first sampling times t k MPC a current state of the driving system is determined based on data collected by sensors of the driving system.

[0048] In a method step V3, a sequence of control signals u is generated by minimizing a given cost function over the time horizon T MPC intended for driving along the trajectory.

[0049] In a process step V4, an optimization problem dependent on the current state is solved.

[0050] In a process step V5, the sequence of control signals u MPC received first control signal u1 MPC at least one actuator Act of the driving system for longitudinal or lateral guidance.

[0051] In a method step V6, the time horizon T is set to the following of the first sampling times t k+1 MPC delay.

[0052] In a process step V7, the repetition of the previous steps is started.

[0053] The time intervals Δt between successive first sampling times t k MPC , t k+1 MPC are each divided into second sampling times t̂ j AI , t̂ i AI , t̂ j AIIn a process step V7, the machine learning model AI, which was trained on data concerning states of the driving system in traffic scenarios, is used to generate control signals û AI for a trajectory control of the driving system, and in the second sampling times t̂ j AI , t̂ i AI , t̂ l AI The second control signals û2 AI certainly.

[0054] In a process step V8, the second control signals û2 AI provided and executed by the Actuator Act.

[0055] In a process step V9, at the first sampling times t k MPC , t k+1 MPC the first control signals u1 MPC with the respective time-parallel second control signals û2 AI compared.

[0056] If the second control signals û2 AIfrom the first control signals u1 MPC deviate significantly, in a process step V10 the first control signals u1 MPC executed by the Actuator Act. Reference symbol MPC model predictive control algorithm AI machine learning model t time A actuator x Location x(t) trajectory x k Point of the trajectory x(t) x k+1 MPC Point of a trajectory from MPC Δ Tolerance range t k MPC first sampling point for MPC Δt time interval T Time horizon u MPC Control signal from MPC u1 MPC first control signal from MPC x i AI Point of a trajectory from AI t i AI second sampling point for AI û AI Control signal from AI û2 AIsecond control signal from AI V1-V10 process step HPU hardware unit CPU1, CPU2 central processor Mem memory AI Acc1, AI Acc2 hardware accelerators for machine learning models X-Bar data transmission device Com comparison unit

Claims

[1] Computer-implemented method for controlling a longitudinal or lateral guidance of a driving system, wherein a model predictive control algorithm (MPC) is used at first sampling times (t k MPC , t k+1 MPC ) is executed and a future trajectory (x k+1 MPC ) of the driving system over a period of time (t k MPC , t k+1 MPC ) divided time horizon (T) and control signals (u MPC ) to realize this trajectory (V1); by at each of the first sampling times (t k MPC ) a current state of the driving system is determined based on data acquired by sensors of the driving system (V2); by minimizing a given cost function over the time horizon (T), a sequence of control signals (u MPC) to drive along the trajectory (V3) by solving an optimization problem dependent on the current state (V4); which is derived from the sequence of control signals (u MPC ) received first control signal (u1 MPC ) is provided to at least one actuator (Act) of the driving system for the longitudinal or lateral guidance (V5); the time horizon (T) is set to the following of the first sampling times (t k+1 MPC ) is moved (V6); the previous steps are repeated, characterized by that the time intervals (Δt) between successive first sampling times (t k MPC , t k+1 MPC ) into second sampling times (t̂ j AI , t̂ i AI , t̂ l AI ); a machine learning model (AI) trained on data concerning states of the driving system in traffic scenarios, control signals (û AI) for a trajectory control of the driving system, and in the second sampling times (t̂ j AI , t̂ i AI , t̂ l AI ) and second control signals (û2 AI ) (V7); the second control signals (û2 AI ) are provided to the actuator (Act) and executed by it (V8); in the first sampling times (t k MPC , t k+1 MPC ) the first control signals (u1 MPC ) with the respective temporally parallel second control signals (û2 AI ) are compared (V9) and if the second control signals (û2 AI ) from the first control signals (u1 MPC ), the first control signals (u1 MPC ) are executed by the actuator (Act) (V10). [2] Method according to claim 1, wherein a distance between two consecutive first sampling times (t k MPC , tk+1 MPC ) corresponds to the time the driving system needs before it enters an unsafe state. [3] Method according to one of the preceding claims, wherein the sequence of the second sampling times (t̂ j AI , t̂ i A1 , t̂ l AI ) relative to the sequence of the first sampling times (t k MPC , t k+1 MPC ) is high frequency. [4] Method according to one of the preceding claims, wherein the longitudinal and transverse guidance of the driving system is controlled. [5] Computer program for controlling a longitudinal or transverse guidance of a driving system, the computer program comprising program instructions which cause the computer to carry out the method according to one of the preceding claims when the computer program is loaded onto the computer or executed by the computer. [6] Hardware unit (HPU) for a driving system comprising at least one central processor (CPU1, CPU2); a memory (Mem); at least one hardware accelerator for machine learning models (AI Acc1, AI Acc2); a data transmission device (X-Bar) via which the central processor (CPU1, CPU2), the memory (Mem) and the hardware accelerator (AI Acc1, AI Acc2) are connected for data transmission; wherein the computer program according to the preceding claim is loadable into the memory (Mem); by means of the data transmission device (X-Bar), when the computer program is loaded, the program instructions of the computer program, which relate to a model predictive control algorithm (MPC) according to the method according to one of claims 1 to 4, are loaded onto the central processor (CPU1, CPU2) or executed by it;by means of the data transmission device (X-Bar), when the computer program is loaded, the program instructions of the computer program relating to a machine learning model (AI) according to the method according to one of claims 1 to 4 are loaded onto the hardware accelerator (AI Acc1, AI Acc2) or executed by it; [7] Hardware unit (HPU) according to claim 6, comprising a plurality of the hardware accelerators (AI Acc1, AI Acc2), wherein the machine learning model (AI) is loaded or executed distributed among the hardware accelerators (AI Acc1, AI Acc2). [8] Use of the computer-implemented method according to one of claims 1 to 4, the computer program according to claim 5 or the hardware unit (HPU) according to one of claims 6 to 7 in an automated operation of a driving system.

Images