Automated driving sequences for off-road vehicles

Abstract

The invention relates to a system for the automated driving control of a vehicle (1) in off-road terrain, which is designed to connect to a sensor unit for detecting terrain and to an actuator unit for adjusting a control variable of the vehicle (1), and to use data about the terrain and a predetermined driving task acquired by the sensor unit as input data for an agent module (3) of the system, which comprises an outer policy (5) which, during its execution, determines a driving strategy with desired vehicle states to solve the driving task, and a plurality of inner policies (7) which are pre-trained for a plurality of different terrain scenarios, wherein the outer policy (5), during its execution, selects one of the inner policies (7) and transmits specifications regarding the desired vehicle states to the selected inner policy (7).

Description

The invention relates to a system for automated driving control of a vehicle off-road in the terrain away from a fixed road, and to a method for producing a system for automated driving control of a vehicle off-road in the terrain away from a fixed road.Off-road driving with motor vehicles away from fixed roads or well-drivable field paths requires the driver in a special way, since some passages can be driven only in a skillful sequence of vehicle actions such as engine / braking torques, steering angles and, if appropriate, the use of off-road actuators such as differential locks, reduction gearing. There is a risk that, in the event of a wrong sequence of such vehicle actions, the vehicle cannot pass, remain stuck, slips or tilts.Methods for supporting maneuver planning of an automatically driving vehicle are known in the prior art. In particular, it is known from the prior art to use adaptive methods, such as a reinforcement learning agent.DE 10 2020 202 350 A1 relates to a method for supporting maneuver planning for an automatically driving vehicle or a robot, wherein a state space of the surroundings of the vehicle or of the robot is described in discrete form by means of a Markov decision problem, wherein optimal action values for discretized actions are determined by means of dynamic programming starting from discrete states in the state space, wherein a mapping with states in the state space as input values and with action values for actions in the state space as output values is learned by means of a reinforcement learning method, wherein a reinforcement learning agent is initialized here on the basis of the optimal action values determined by means of dynamic programming, and wherein the learned mapping is provided for maneuver planning.Furthermore, DE 10 2023 200 080 A1 shows a method and control device for operating a vehicle, wherein a set of trajectory candidates for fulfilling a driving task of the vehicle is generated using a deep learning-based trajectory generator, the trajectory candidates are read in by a cost-based trajectory selector and are evaluated with a cost value based on predefined criteria, wherein the trajectory candidate with the best cost value is selected from the set and is transferred to a vehicle controller of the vehicle for execution. In addition, DE 10 2021 211 152 B3 relates to an apparatus and a computer-implemented method for providing training data to an agent for reinforcement learning for generating a strategy designed for controlling a vehicle, to a corresponding apparatus and a method for generating a strategy designed for controlling a vehicle, and to a vehicle. DE 10 2017 200 580 A1 concerns a method for optimizing maneuver planning for autonomously driving vehicles.It is an object of the invention to improve the automatic driving of a vehicle on terrain without a road.The invention results from the features of the independent claims. Advantageous refinements and refinements are the subject matter of the dependent claims.A first aspect of the invention relates to a system for automated driving control of a vehicle on the terrain away from a fixed road, wherein the system is designed for connection to a sensor unit for detecting a terrain in the environment of the vehicle and for connection to an actuator unit for setting a predefined manipulated variable of the vehicle that changes a vehicle state, and for using data about the terrain detected by the sensor unit and a predefined driving task as input data of an agent module of the system, wherein the agent module comprises an external policy implemented in one or more pre-trained models of the machine learning, which external policy, when executed, determines a driving strategy with desired vehicle states for solving the driving task, and wherein the agent module implements a plurality of data about the terrain detected in one or more pre-trained models of the machine learning, The device according to the invention comprises internal policies which are pre-trained for a plurality of different terrain scenarios and each implement the vehicle states predefined by the external policy by determining and predefined actuating variables when they are executed, and wherein the external policy selects one of the internal policies when they are executed and transmits predefined values for the desired vehicle states to the selected one of the internal policies, wherein the agent module is designed to transmit the predefined values for actuating variables generated by the respectively selected internal policy to the actuator unit as an actuator command.The outer policy and the inner policies are the result of a training process of reinforcement learning, the so-called reinforcement learning. These are realized via models of machine learning, in particular artificial neural networks. The artificial neural networks have a high variety of internal parameters that are the result of the training process. Preferably, a single model of machine learning, i.e. in particular a single artificial neural network, is used for the external policy. In the internal policies, on the one hand, a large and very deep artificial neural network can be used to implement the various internal policies. However, the internal policies can also be divided into respective separate artificial neural networks. A hierarchical architecture is used in this case:The external policy is responsible for presetting driving conditions for a given driving task. An object of the driving is, for example, the traversing of a beach, the driving of a staircase, a corollary, a sandy subgrade, etc.; in this case, on the one hand, target specifications are to be taken into account, for example the speed, safety, avoidance of high loads on the vehicle, a high driving comfort; a high safety is achieved if rolling over or tipping over of the vehicle is prevented with a safety margin, just as the vehicle remains stuck. These are therefore also precisely the goals that are applied in pretraining the models of machine learning in a reward function of reinforcement learning. The external policy therefore initially analyzes a data record about the environment of the vehicle accordingly and determines a driving strategy. However, individual driving states are in turn themselves setpoint variables, such as, for example, a speed of the vehicle, a yaw rate of the vehicle, an acceleration of the vehicle with respect to the underlying surface, i.e., kinematic states are defined in particular as vehicle states. A drive torque or a braking torque, as well as a steering angle or a chassis setting, is a control variable which is generated by an actuator unit of the vehicle.The internal policies are pre-trained specifically for certain environmental properties. For example, a first of the inner policies is trained for sandy subsurface, a second of the inner policies for flux runs, a third of the inner policies for steep ramps in curves, a fourth of the inner policies for connecting high obstacles. The hierarchical approach provides that the outer policy therefore takes over in particular the trajectory planning, i.e. a navigation task, and selects for a driving task one of the inner policies such that the most suitable of the inner policies for a present terrain is selected as far as possible. If, for example, a sandy environment is detected by the agent module, one of the inner policies that has been pre-trained for a sandy environment is selected by the outer policy. In addition to the selection of the inner policies, the task of the outer policy is to transmit a target vehicle state as a specification to the selected one of the inner policies. For example, a sandy section should be traveled at a specific speed and a specific curve rate without the vehicle being stuck. This selected inner policy implements the command of the outer policy as optimally and successfully as possible. For this purpose, it selects, for example, corresponding wheel torques, time derivatives of these and braking torques in order to prevent, as far as possible, sticking and to be able to implement the desired trajectory of the driving task.The driving task is determined in particular by means of a starting point and a destination, while the specific implementation of the driving around a path between the starting point and the destination is left to the agent module within certain limits.In this case, the agent module uses an adaptive approach which variably determines the selection of the internal policies to be applied to the corresponding specifications of the driving state as a function of environmental properties, in particular properties of the underlying surface. To ascertain data about the environment of the vehicle, in particular its background, data of a sensor unit of the vehicle are used, which is designed in particular for detecting a terrain in the environment of the vehicle.Alternatively, it may be sufficient in principle to determine the current location of the vehicle in the presence of a sufficiently accurate and high-resolution digital map and to obtain information about the environment from the map data. The aim in each case is to obtain a map with elevation information in order to be able to achieve optimum path planning by the external policy of the agent module.However, the inner policies can also be used already in the path planning in order to be able to determine an optimum path even before the section for performing the driving task is traveled on. In this case, the inner policies, at least a part of them, can be executed by the outer policy and the results are compared with one another, after which one of the inner policies which is optimum for the present driving task is used for the actual implementation of the driving task.For this purpose, a sensor unit on the vehicle is preferably used, which is designed for depth measurement, in particular comprising lidar, radar, ultrasound or a camera with depth information (RGB-D). Particularly high sensor positions, for example on the roof of the vehicle, are optionally preferred here in order to improve a view of the sensor unit into the environment; however, sensor positions which are directed under or in front of / behind the vehicle can also be used.A geometric map, which comprises the surface information in the sense of a geometric profile and a surface quality, can be determined on site with the aid of this sensor unit in order to obtain current information. This has the advantage over the use of a digital map from a memory that current environmental information is used in each case-for example, after an unauthorized person a branch may be completely changed with respect to the stored data of a digital map. If a destination is defined and a starting location of the vehicle of an in particular current point of view is selected, the essential part of the driving task for reaching the destination is already fixed.For optimum navigation of the vehicle through the agent module, the depicted terrain is used, but stored capabilities of the vehicle are also used, such as the capability for overcoming a maximum forward / rearward gradient, maximum transverse inclination, maximum step height, a smallest curve radius, a maximum ground clearance of subfloor or else overhang forward / rearward, a maximum material load / twist.A reward function may also be calculated during travel without or with the models of machine learning being adjusted. The reward function can be used, for example, to abort the execution of the policies and request manual driver override if the reward function falls below a predefined threshold value.The driving task may also be divided into individual sections, each of which is analyzed by the agent module and a partial driving task is implemented. The sections are preferably reset in the event of substantial changes in the properties of the traveled terrain, such that a section exactly depicts a contiguous property of the traveled terrain for which an optimum one of the internal policies can be selected, and a recalculation of the navigation can be carried out by the external policy. In this case, a recursive approach can also be used in which the outer policy first specifies an entire path over all sections, and individual strategies with selected vehicle states within the sections can again be executed over the sections by executing the hierarchy again.According to an advantageous embodiment, the system is designed to generate a geometric and / or semantic map from the data about the terrain as a digital image of the terrain and to transmit it to the agent module, which, when executed, determines the desired vehicle states by path planning with the aid of the map, wherein the path planning extends from a starting location defined on the map to a destination location defined on the map.This sensor data about the environment of the sensor unit has, in particular, a point cloud which allows the mapping into a geometric map. A transformation of the data into a bird's eye view looking from above onto the vehicle or the environment around the vehicle may be generated. Preferably, height information is determined for each coordinate in a plane and assigned to the coordinate.According to a further advantageous embodiment, the system is designed to use a current position of the vehicle on the map, a current vehicle state, and terrain properties derived from the map as inputs of the external policy.The terrain properties comprise in particular information about the geometric surface such as the gradient, transverse inclination, step height, unevenness, tread size (a description of the flat fit of a tire on an elevation), but can also comprise qualitative properties such as an estimated coefficient of friction, which can be estimated, for example, using methods of semantic segmentation.According to a further advantageous embodiment, the external policy determines, when it is used, a similarity measure between current data about the terrain in the environment of the vehicle and terrain data used for pretraining the respective internal policies and selects one of the internal policies according to the respective similarity measure.Such a similarity measure is formed, for example, from a quadratic distance of height profiles, slopes, terrain properties. The selection of the inner policies via a similarity measure may be performed by a machine-learned mapping. In this case, predetermined pairs of input variables and output variables can be stored in a database during the training process thereof. This mapping can then be trained by means of supervised learning.According to a further advantageous embodiment, the agent module is designed to adapt the external policy and / or the internal policies by reinforcement learning using a definition for a weighted reward function during and / or after the journey of the vehicle.The reward function is preferably predefined and is composed of individual terms, preferably driving comfort, stability of the vehicle, low component load, low controller activity and speed when executing the driving task. While on the one hand an exploration can take place during the handling of the driving task and by varying the vehicle states and manipulated variables and an adaptation of the models of the machine learning in this case, a predictive simulation can also be carried out before a section of the driving task to be handled is being used and the models of the machine learning can be adapted in the simulation.According to a further advantageous embodiment, the agent module is designed to vary the desired vehicle states and / or predefined manipulated variables within predefined limits during the travel of the vehicle and to adapt the desired vehicle states and / or predefined manipulated variables as a function of the change in the reward function associated therewith.According to a further advantageous embodiment, the agent module is designed to likewise adapt parameters of a respective model of the machine learning of the external policy and of the internal policies with the adapted vehicle states and / or predefined manipulated variables.According to a further advantageous embodiment, the agent module stores a multiplicity of restrictions of vehicle states which are taken into account in the determination of the desired vehicle states and specifications of the manipulated variables by the external policy and the internal policies during their execution.Such restrictions may include certain characteristics of the vehicle, such as maximum climbing ability, step height, tire type, ride height of the vehicle.A further aspect of the invention relates to a method for producing a system for automated driving control of a vehicle on the terrain away from a fixed road, wherein the system is provided for connection to a sensor unit for detecting a terrain in the environment of the vehicle and for connection to an actuator unit for setting a predefined manipulated variable of the vehicle which changes a vehicle state, and for using data about the terrain detected by the sensor unit and a predefined driving task as input data of an agent module of the system, wherein an external policy implemented in one or more models of machine learning is used for the agent module, and a plurality of internal policies implemented in one or more models of machine learning are used for the agent module, wherein the models of machine learning are trained with the aid of a predefined reward function by reinforcement learning in a simulation with a multiplicity of virtual scenarios to be traveled, which each represent a terrain away from a fixed road, wherein the respective inner policies are trained for a multiplicity of different terrain scenarios, and the outer policy and the inner policies are brought into a hierarchical link with one another such that a respective one of the inner policies, when executed, converts the vehicle states predefined by the outer policy by ascertaining and predefined manipulated variables and specifies said vehicle states to the actuator unit as a respective actuator command, and that the external policy, when executed, determines a driving strategy with desired vehicle states for solving the driving task and selects one of the internal policies and transmits specifications on the desired vehicle states to the selected one of the internal policies.In this method, a method of reinforcement learning is applied in a plurality of simulation passes without necessarily applying supervised learning. Rather, a respective reward function is evaluated by independent variation of manipulated variables and predefined vehicle states and, due to the large plurality of simulations with varied parameters, it is optimized, so that the reward function is maximized. In this case, both the models of the inner policies and the model of the outer policy, preferably all respective artificial neural networks, are adapted accordingly. This creates a possibility of how sequences of vehicle actions can be learned to master difficult off-road passages. In the simulation, for the broad applicability of the agent module, correspondingly multiple scenarios are specified in order to train the specialized internal policies in particular. Thus, different terrain properties can be predefined, and in particular a multiplicity of different characteristics within a category of terrain properties can be simulated. In contrast to the real driving operation, an extremely high number of different combinations of boundary conditions can be simulated in the simulation, i.e. of different vehicle states, weather conditions, environmental conditions and driving tasks.Different driving tasks can also be generated by varying different starting points and destinations. In principle, real data can be used, which are used in the simulation; in addition or alternatively, synthetic data can be introduced. The simulation and reinforcement learning of the models of machine learning can be ended if predefined conditions are fulfilled, for example the training time or the number of iterations reach a predefined limit, the value of the reward function falls below a predefined threshold value, or if the relative change over further simulation passes falls below a limit value, i.e. an improvement of the agent module is only small. Advantageously, off-road properties are varied within a range of values when generating the geometric map. This can be done randomly, normally or Gaussianally distributed, inter alia by domain randomisation. Furthermore, methods known in the prior art can be used for training one or more models of machine learning, such as, for example, the so-called "domain adaptation", or the generation of a "residual model".According to a further advantageous embodiment, the models of machine learning are trained with the aid of a predefined reward function by reinforcement learning in a simulation with a multiplicity of artificial scenarios to be traversed, which each represent a fixed road.Advantages and preferred refinements of the proposed method result from an analogous and analogous transfer of the statements made above in connection with the proposed system.Further advantages, features and details are evident from the following description, in which--possibly with reference to the drawing--at least one exemplary embodiment is described in detail.The following are shown: FIG. 1 : A vehicle with a system for automated driving control in the terrain according to an exemplary embodiment of the invention. FIG. 2 : shows an architecture of the system of FIG. 1.FIG. 1 shows a vehicle 1 having a system for automatically controlling the driving of the vehicle 1 off-road in the terrain away from a fixed road. The vehicle 1 is designed structurally for terrain and has a corresponding ground clearance and a corresponding drive train. With the aid of the system, a driver of the vehicle 1 is able to reach a destination in an automated manner, which destination can preferably be defined by the user by being specified at an input unit of the vehicle 1. A sensor unit detects a surrounding area of the vehicle 1, which stands in front of a difficult passage. This passage has mixed terrain comprising a solid but hilly subgrade initially with a transition to a fluid bed with soft, sludgey subgrade and partly deeper water. Upon activation of the system in the vehicle 1, the vehicle 1 executes an agent module that includes a pre-trained outer policy 5 and a plurality of inner pre-trained policies 7. An architecture of the hierarchical configuration of these policies is shown and explained in more detail in Fig. 2. The agent module has been pretrained by a method of reinforcement learning and contains corresponding parameters in neural networks. The outer policy 5 performs a commanding navigation function that determines an optimal path through the road ahead to the destination. This optimal path is determined with the aid of a map which is generated from the data of the sensor unit of the vehicle 1. If the path is fixed, the best one of the inner policies 7 is selected depending on the individual conditions of the terrain including the slopes as well as on estimated frictions and an estimated penetration depth of the vehicle 1. The inner policies 7 are pre-trained to specific subsurface conditions and can thus convert optimum vehicle states, which are predefined by the outer policy 5, into actuating variables for an actuator unit of the vehicle 1. The outer policy 5 has to take into account which water depth is estimated so that the vehicle 1 does not become too fast when passing, since otherwise a bow wave can form and the maximum water depth can be traveled. In this case, floating could occur until the air in the vehicle floor has leaked. In the meantime, the vehicle 1 could be flushed or twisted and depart from the desired path. Accordingly, the outer policy 5 specifies vehicle conditions such as a desired speed, a desired steering angle, and a desired entry angle into the body of water. For this purpose, one of the inner policies 7 is first selected, which has been specifically trained for traversing bodies of water. A corresponding smooth braking torque of the wheels and torque of the drive train is selected by this of the inner policies 7 in order to remain with optimum slip even in the case of a sludgey subgrade on the optimum path determined by the outer policy 5. In this case, the agent module continuously determines a reward function as to how well the implementation of this driving task for traversing the water body is carried out. Both the outer policy 5 and the inner policies 7 can be adapted within certain limits in order to keep the reward function favorable. If, for example, a desired speed is exceeded, the torque is not only increased by control technology, but also parameters of the activated one of the inner policies 7 are adapted in order to already enable an improved approach in the command, i.e. feed-forward signal. A similar strategy may be applied to the agent module when the vehicle 1 is stationary in order to clear the vehicle 1 and further improve the agent module.FIG. 2 schematically shows the architecture of the agent module 3. Shown herein are the inner policies 7 mentioned in Fig. 1, only one of which is referred to as such for simplicity. For each section of a desired route to be covered, wherein a respective section is characterized by a certain route and / or travel duration and / or quality of the terrain, one of the inner policies 7 that is as optimal as possible is selected. This selection is made by the outer policy 5, which transfers desired driving states as desired specifications analogously to a cascade controller of the selected one of the inner policies 7, which in turn sets target speeds, target slip, target wheel angle, etc. by corresponding control variables. The manipulated variables must be selected accordingly adaptively depending on the predefined terrain, for example because different terrains can have completely different coefficients of friction. Such an architecture of the agent module is preferably generated in two steps. First, in a simulation environment, the repeated and as corrective as possible training of the inner policies 7 takes place for their respective special terrain scenarios, then the outer policy 5 is trained, wherein the function of the inner policies 7 can already be taken into account during the training in order to be able to achieve as optimal an overall effect of the agent module as possible.Although the invention has been illustrated and explained in more detail by preferred exemplary embodiments, the invention is not restricted by the disclosed examples and other variations can be derived therefrom by the person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a large number of possible variations exist. It is also clear that embodiments mentioned by way of example represent only examples which are not to be understood in any way as limiting, for example, the scope of protection, the possible applications or the configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, wherein the person skilled in the art, knowing the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.List of reference characters1 Vehicle 3 Agent module 5 Outer policy 7 Inner policies

Claims

System for automated driving control of a vehicle (1) on the terrain away from a fixed road, wherein the system is designed for connection to a sensor unit for detecting a terrain in the environment of the vehicle (1) and for connection to an actuator unit for setting a predefined manipulated variable of the vehicle (1) which changes a vehicle state, and for using data about the terrain detected by the sensor unit and a predefined driving task as input data of an agent module (3) of the system, wherein the agent module (3) comprises an external policy (5) implemented in one or more pre-trained models of the machine learning, which external policy, when executed, determines a driving strategy with desired vehicle states for solving the driving task, and wherein the agent module (3) comprises a plurality of, The device according to the invention comprises internal policies (7) implemented in one or more pre-trained models of machine learning, which are pre-trained for a plurality of different terrain scenarios and each implement, when executed, the vehicle states predefined by the external policy by ascertaining and predefined actuating variables, and wherein the external policy (5) selects, when executed, one of the internal policies (7) and transmits, as an actuator command, specifications to the desired vehicle states to the selected one of the internal policies (7), wherein the agent module (3) is designed to transmit the specifications generated by the respectively selected internal policy to actuating variables to the actuator unit.The system according to claim 1, wherein the system is configured to generate a geometric and / or semantic map from the data about the terrain as a digital image of the terrain and to transmit it to the agent module (3), which, when executed, determines the desired vehicle states by path planning with the aid of the map, wherein the path planning extends from a starting location defined on the map to a destination location defined on the map.The system according to claim 2, wherein the system is adapted to use a current position of the vehicle (1) on the map, a current vehicle condition, terrain characteristics derived from the map as inputs of the external policy.The system according to any one of the preceding claims, wherein the external policy (5), when applied, determines a similarity measure between current data about the terrain in the environment of the vehicle (1) and terrain data used for pretraining the respective internal policies (7) and selects one of the internal policies (7) according to the respective similarity measure.The system according to any of the preceding claims, wherein the agent module (3) is configured to adapt the outer policy (5) and / or the inner policies (7) by reinforcement learning using a definition for a weighted reward function during and / or after the travel of the vehicle (1).The system according to claim 5, wherein the agent module (3) is designed to vary the desired vehicle states and / or predefined control variables within predefined limits during the travel of the vehicle (1) and to adapt the desired vehicle states and / or predefined control variables depending on the change in the reward function associated therewith.The system according to claim 6, wherein the agent module (3) is configured to also adapt parameters of a respective model of machine learning of the outer policy and the inner policies (7) with the adapted vehicle states and / or predetermined manipulated variables.System according to one of the preceding claims, wherein the agent module (3) keeps stored a multiplicity of restrictions of vehicle states which are taken into account in the determination of the desired vehicle states and specifications of the actuating variables by the external policy (5) and the internal policies (7) during their execution.Method for producing a system for automated driving control of a vehicle (1) on the terrain away from a fixed road, wherein the system is provided for connection to a sensor unit for detecting a terrain in the environment of the vehicle (1) and for connection to an actuator unit for setting a predefined manipulated variable of the vehicle (1) which changes a vehicle state, and is provided for using data about the terrain detected by the sensor unit and a predefined driving task as input data of an agent module (3) of the system, wherein an external policy (5) implemented in one or more models of machine learning is used for the agent module (3), and a plurality of internal policies (7) implemented in one or more models of machine learning are used for the agent module (3), wherein the models of machine learning are trained with the aid of a predefined reward function by reinforcement learning in a simulation with a multiplicity of virtual scenarios to be traveled, which each represent a terrain away from a fixed road, wherein the respective inner policies (7) are trained for a multiplicity of different terrain scenarios, and the outer policy (5) and the inner policies are brought into a hierarchical linkage with one another such that a respective one of the inner policies (7) upon execution thereof converts the vehicle states predefined by the outer policy by ascertaining and specifying manipulated variables and specifies said vehicle states to the actuator unit as a respective actuator command, and that the external policy (5) determines a driving strategy with desired vehicle states for solving the driving task when it is executed and selects one of the internal policies (7) and transmits specifications on the desired vehicle states to the selected one of the internal policies (7).The method of claim 9, wherein the models of machine learning are trained using a predetermined reward function by reinforcement learning in a simulation with a plurality of artificial scenarios to be traversed, each mapping a fixed road.

Images