Method for adapting and evaluating a control device comprising a policy artificial neural network, and control device comprising a policy artificial neural network

Abstract

The invention relates to a computer-implemented method for adapting a control device, which is configured to detect a current state and to determine an action on the basis of the current state by means of a policy artificial neural network (policy ANN), by means of a critic artificial neural network (critic ANN). The policy ANN and the critic ANN are adapted according to a restricted reinforcement learning method. A barrier term which ensures that the input / output relationship of the policy ANN complies with a Lipschitz condition with a predetermined upper limit for the Lipschitz constant L Bnd is included in the adaptation of the policy ANN. This guarantees robustness of the policy ANN with respect to input disturbances. The invention additionally relates to a computer-implemented method for evaluating the robustness of a control device and to a control device comprising a policy ANN which has been adapted and / or evaluated using an aforementioned method.

Description

[0001] Mercedes-Benz Group AG Method for adapting and evaluating a control device comprising a policy artificial neural network and control device comprising a policy artificial neural network. The invention relates to a method for adapting a control device that is used to detect a current state and to determine an action based on the current state by means of a policy artificial neural network (policy ANN). ^ ^ is set up. Furthermore, the invention relates to a method for evaluating such a control device. In addition, the invention relates to a control device that is a control device. ^ Policy KNN adapted or evaluated using such a procedure is included. ^^Reinforcement learning methods comprise an agent and a description of an environment. The agent is intended to solve a problem implicitly formulated by the environment. The environment is observable by a generally multidimensional vector of state parameters. For example, in the case of an inverted pendulum that can pivot along a single axis of rotation, the environment can be described by a rotation angle (displacement from the neutral position, in which the pendulum stands upright against the direction of gravity) and by an angular velocity. ^Starting from an initial state, a subsequent state is influenced by an action performed by the agent. For example, a new angle of rotation and a new angular velocity of the inverted pendulum are influenced by a torque applied to its axis of rotation. Furthermore, the subsequent state is influenced by disturbances in a non-deterministic manner and / or results from the initial state in a non-deterministic manner. According to a policy, the agent selects an action from a set of possible actions, which are bounded by a feasible action space. A policy assigns an action and a probability measure for that action to a currently observed state. The instantaneous success of an action is determined by a reward value, for example, as the changed displacement of the inverted pendulum after applying the torque to its axis of rotation.The quality (or value) of an action is determined by the expected future sum of reward values, which can be estimated, for example, as the discounted sum of rewards. The agent's actions should be designed to maximize this long-term achievable sum of rewards. The agent can choose an action deterministically or non-deterministically (stochastically). An action chosen stochastically by the agent is statistically determined (according to its assigned probability measures). The environment in which an agent acts by choosing actions can be described as a sequence of states characterized by transition probabilities, similar to a Markov decision process.Starting from an initial state, this sequence (i.e., the transition probabilities) is determined by the agent's actions (deterministically or stochastically) selected according to a policy, the (potentially stochastic) system dynamics, and potentially by non-deterministic disturbances. The learning problem lies in selecting a policy that, in a static sense, defines the best possible actions. In other words, the expected value of the sum of reward values ​​achieved over a sufficiently large number of steps should be maximized by the selected policy. Reinforcement learning methods, referred to as "actor"-"critic" methods, form separate structures: an actor and a critical.The Actor implements a policy (that is, the selection of an action depending on an observed current state), which can be parameterized using a generally multidimensional policy parameter vector. The Critic approximates or estimates the action value (as a discounted sum of rewards) of an action. The Critic estimates the discounted sum of rewards that can be expected, starting from the current state of the environment and a defined initial action, if the agent were to select further actions based on the current policy. The Critic thus depends on both the behavior of the environment and the agent's current policy. The Critic can also be parameterized using a generally multidimensional Critic parameter vector. In particular, see, for example, the publication by RS Sutton and AG Barto, "Reinforcement Learning: An Introduction," Cambridge, MA, USA.Bradford Book, 2018, describes methods in which both the actor and the critical are implemented as artificial neural networks (ANNs). The reinforcement learning problem comprises, on the one hand (in a first problem layer), procedures for optimizing and evaluating the policy based on a policy or value function, which can each be implemented as a parametric function using an ANN. On the other hand (in a second problem layer), the reinforcement learning problem involves approximating the policy or value function by adjusting the parameters of the respective ANN. This second problem layer can then be reduced to the usual training of ANNs, in which their weights are adjusted to minimize a learning error, for example, using a gradient descent method related to the learning error (in supervised learning) or to another performance criterion.The weights of the policy KNN (corresponding to the policy parameters) are determined such that an optimality criterion with respect to the policy parameters is maximized. This optimality criterion is determined by the (discounted) sum of the reward values ​​assigned to subsequent states depending on the actions selected according to a policy. Therefore, one could also say that the optimality criterion corresponds to the expected reward determined across the entire state space. The weights of the critical KNN (corresponding to the critical parameters) are determined such that a statistical error measure (for example, mean squared deviation) is minimized. In a learning method known as temporal difference learning, a current action value is determined based on an estimated action value of the subsequent state and on a reward derived from the transition from the current to the subsequent state.For example, a current (unknown) action value ^^(^, ^) for a current state ^ and an action ^ determined according to a policy ^ can be determined by making an estimate. des The action value of the subsequent state ^′ is determined, assuming that in the subsequent state ^^ actions ^′ are selected according to the current policy ^, where a reward ^′ has been assigned to the state transition from ^ to ^′: Actual reward values ​​of a transition from one state ^ to a subsequent state ^ ^The reward values ​​are unknown and must be estimated using a policy evaluation procedure. For example, an actual reward value can be estimated using a reward value estimated for a subsequent state and a current reward value. Estimated reward values ​​determined by the Critic KNN only capture the consequences of a selected action incompletely and approximately. Once the parameters of the Policy KNN and the Critic KNN have been determined through training, a sensitivity can be assigned to each of their respective input-output functions. The sensitivity is defined as a measure of how much the KNN's output changes when its input changes. The sensitivity can be given as a local approximation using the Jacobian matrix, which captures the partial derivatives of the output values ​​(present at the KNN's output neurons) with respect to the input values ​​(present at the KNN's input neurons).Sensitivity describes, in particular, the change in a policy KNN (k-nearest neighbors) algorithm due to deviations in input values. These deviations can occur, for example, due to measurement errors when capturing actual values ​​that characterize the state of the environment, or due to nonlinearities in an actuator or a controlled system. Low sensitivity is especially desirable for policy KNNs, as it corresponds to low susceptibility to disturbances or high robustness. In other words, policy KNNs with lower sensitivity trigger less pronounced deviations or oscillations in the output when their input is perturbed or disturbed. They are therefore preferred for achieving stable and efficient control of an environment.Since the Jacobian matrix is ​​determined by the respective input values ​​of the KNN, determining the sensitivity based on the Jacobian matrix is ​​neither numerically nor analytically practical for investigating robustness, as this would require determining the Jacobian matrix for a wide variety of operating points (i.e., input neuron assignments). In particular, determining guaranteed robustness in the sense of maximum sensitivity is only possible with an infinite number of operating points randomly selected, for example, using a Monte Carlo method. Therefore, there is a need for methods that can estimate the sensitivity of a policy KNN and / or a critical KNN and ideally control it during training. (See the publication by M. Fazlyab, A. Robey, H. Hassani, M. Morari, and GJ.)Pappas, "Efficient and accurate estimation of lipschitz constants for deep neural networks" in Advances in Neural Information Processing Systems (NeurIPS), 2019 describes a method for determining a guaranteed upper bound for the Lipschitz constant that can be assigned to a (trained) KNN. A Lipschitz constant is understood to be the non-negative number ^^ that, depending on a norm of a difference of (generally multidimensional) input values ​​−^^, limits a difference of (generally multidimensional) output values ​​determined by a function ^: ℝ^^ → ℝ^^ with input dimension (dimension of the domain) ^^ and output dimension (dimension of the range) ^^: ‖^(^^) − ^(^^)‖^ ≤ ^^‖^^ − ^^‖^ Such a function ^: ℝ^^ → ℝ^^ can, for example, be implemented by a KNN with ^^ input neurons and ^. ^Output neurons are implemented. Publication US 2021 / 0110271 A1 describes methods, systems, and devices for training KNNs configured to select an agent's action according to a policy (policy KNNs). The policy KNN is used to select actions for an agent interacting with an environment by receiving an observation that characterizes a state of the environment and by selecting an action from a set of possible actions depending on the observed state. A trajectory is captured from a replayable memory. A final update of the current values ​​of the policy KNN's parameters is determined for each observation in the trajectory used to train the policy KNN. The final updates are determined based on selected action updates and leave-one-out updates.The invention is based on the objective of providing an improved method for adapting a control device that is configured to detect a current state and determine an action based on the current state using a policy KNN. In particular, an objective of the invention is to provide a method by which a policy KNN can be adapted in such a way that a predetermined robustness is guaranteed. This objective is achieved according to the invention with a method having the features of claim 1. The invention is further based on the objective of providing an improved method for evaluating the robustness of a control device that is configured to detect a current state and determine an action based on the current state mean of a policy KNN.In particular, one object of the invention is to provide a method for determining a particularly sharp (narrow) boundary with regard to guaranteed robustness. This object is achieved according to the invention with a method having the features of claim 4. The invention also aims to provide an improved control device that is configured to detect a current state and determine an action based on the current state using a policy KNN. In particular, one object of the invention is to provide a control device that guarantees a predetermined robustness in its input-output behavior. This object is achieved according to the invention with a control device having the features of claim 5. Advantageous embodiments of the invention are the subject of the dependent claims.The invention is based on the finding that the slope of an activation function^(^) of a neuron in a KNN (regardless of whether it is a policy KNN or a critical KNN) can be restricted to an interval[^, ^] with 0 ≤ ^ < ^ < ∞. That is: where ^^ , ^^ are each input values ​​on the same scalar activation function ^: ^ → ^, which is implemented by a neuron of the KNN. Based on this finding, the determination of the tightest possible estimate for the Lipschitz constant can be achieved by solving the restricted optimization problem m, which can be considered a semidefinite program. ^ ^ ^ in ^where ^ (^ ^ , ^, ^) ≼ 0 (negative semidefinite) ^,^ ^ ^ where the symmetric matrix ^ (hereinafter also referred to as the Lipschitz matrix) is given by: where ^ is a positive semidefinite diagonal matrix with the elements ^^^ , ^ = 1, … ^ (^ is the total number of all neurons of the policy KNN, ^ = In particular, the invention is based on the finding that if at least one matrix ^ exists that satisfies this constraint, then ^^ is a Lipschitz constant of the KNN (that is: its input-output function ^: ℝ^^ → ℝ^^) is bounded by a Lipschitz constant ^^ ∈ ^^^). Here, the matrix ^ is a block diagonal matrix comprising the weight matrices of the connections between adjacent layers of the KNN: where ^^ is the matrix of connection weights from the ^-th to the ^ + 1-th layer of the KNN, each comprising ^^, ^^ , … ^^ … ^^ neurons (that is: ^^ is the input dimension and ^^ is the output dimension of the KNN) and where the matrix ^ is composed of unit matrices arranged in a band-like fashion in a right-diagonal submatrix Since both the optimization criterion and the constraint ^ (^ ^^ , ^, ^) ≼ Since the respective decision variables are linear, a linear (and therefore convex) optimization problem is obtained, which is particularly easy to solve. The robustness of the policy KNN can be evaluated using the Lipschitz constant thus determined. Furthermore, the invention is based on the insight that a particularly robust (i.e., one that is particularly insensitive to disturbances of its input values) policy KNN can be obtained by formulating a constraint during its training, in addition to a conventional optimization criterion, which limits its Lipschitz constant. When training the policy KNN with known methods, a parameter vector of policy parameters (which comprises the entirety of the adjustable weights of the policy KNN) is sought such that an optimization criterion m^ax ^ (^ ^)The objective function ^ (^ ^) is maximized with respect to the policy ^ ^ that the policy KNN implements (depending on the set of its parameters ^). In other words, the policy KNN assigns an action (e.g., a torque applied to the axis of rotation of the inverted pendulum) to each state presented as a training example at its input (e.g., the angle and angular velocity of an inverted pendulum), which is evaluated based on the discounted sum of future expected reward values. The policy KNN is trained such that the appropriately weighted reward values ​​collected from the set of all training examples are maximized. A policy KNN trained in this way (that is, a parameter vector ^ determined in this way) can have its sensitivity bounded by a Lipschitz constant if the activation functions of the neurons are bounded in their slopes.If any positive semidefinite (diagonal) matrix ^ of diagonal elements ^^^ ≥ 0, ^ =1 … ^ exists, for which, together with the parameter vector ^, the Lipschitz matrix ^ is negative semidefinite: ^. (^ ^ If ^, ^, ^) ≼ 0, then ^^ is a Lipschitz constant for the policy KNN. The diagonal elements ^^^ ≥ 0 are subsequently also represented in a diagonal vector ^ ∈ ℝ ^ ^ ^ In summary, considering this relationship, the training of the policy KNN is modified according to the invention such that in an optimization problem (now also including the matrix ^) the constraint ^ ^^ ^ ^^^ , ^, ^^ ≼ 0 is forced (hereinafter referred to as the "modified optimization problem"). The parameters of the policy KNN summarized in the parameter vector ^, together with the positive semidefinite diagonal matrix ^, are to be optimized such that the optimality metric ^ (^ ^) is maximized and yet simultaneously the Lipschitz matrix ^ ^^ ^^^^ , ^, ^^ remains negative semidefinite. In other words, a training is forced that uses a policy KNN with a Lipschitz constant of at most ^ ^^^ generated. Thus, depending on the given limit ^ ^^^ , guarantee a predetermined robustness. Considering the constraint ^ ^^ ^^^^ , ^, ^^ ≼ 0 (in which the parameters ^ and ^ are bilinearly linked and in which the values ​​of ^ appear quadratically) when optimizing the objective function ^ (^ ^) thus leads to an optimization problem that is nonlinear in the decision variables and in which the set of feasible solutions formed by ^ and ^, bounded by the constraint, is non-convex. According to a first formulation, a solution to the modified optimization problem can be found by performing a training step with respect to the original (unbounded) optimization criterion m ^ax ^ (^ ^)The decision variable ^ (comprising all network weights of the policy KNN) is restricted to the set of permissible values. In other words, the optimization problem is first solved without considering the constraint, and an update candidate for the decision variable ^ is determined that could solve the unrestricted optimization problem. Then, the projection of the update candidate ^^ onto the decision variable ^ that minimizes the squared Euclidean distance to ^ ^ is determined. and which at the same time satisfies the condition that the matrix determined by equation (1) is negative semidefinite ^ ^^ ^^^^ , ^, ^^ ≼ 0 For a particularly simple solution to this projection problem, the constraint can be transformed into a convex representation as explained below. This makes a solution possible using methods of linear programming. This applies to many common activation functions of KNNs, for example, for a "rectified linear unit" (ReLU) activation function ^(^) = max(0, ^) or for a hyperbolic tangent ^(^) = tanh(^) or for a sigmoid activation function. ^(^) = ^The minimum slope can be set to ^ = 0, which reduces the constraint, using Schur's complement, to: The invention is based on the finding that if one of the decision variables is considered constant, the matrix inequality (2) becomes linear in the remaining decision variable, and the resulting restricted set of solutions becomes convex. The invention therefore proposes, instead of optimizing simultaneously for both decision variables (with a generally non-convex solution set), to optimize stepwise (sequentially) for each of the decision variables at a time, thereby decomposing the problem into convex subproblems, each of which can be described as a quadratic optimization problem with a constraint given as a linear matrix inequality.In one embodiment, a reformulation of the optimization problem is proposed in which the original constraint imposed by the matrix inequality ^ ^^ ^^^^ , ^, ^^ ≼ 0 is captured by a logarithmic barrier term formed over the negative determinant of the (negative semidefinite) matrix ^ ^^ ^^^^ , ^, ^^:. Such an (unrestricted) optimization problem can be solved, for example, using standard gradient methods of unrestricted numerical optimization, where the gradient ∇^ ^(^^) from the original (unrestricted) optimization problem is already known. The gradient of the barrier term with respect to the parameter vector ^ is to be calculated as follows (for improved readability, ^^^ is used here and below). ^ ^^^ , ^, ^^ (abbreviated with ^):∇ ^ log d ( ) ( ) ^^^ ^^ ^et −^ = ^ ^det −^ ^^ ^^^ ^^ ^ ^^^,…,^ ^Furthermore, the gradient of the barrier term with respect to the diagonal elements is ^ ^^ To calculate the matrix ^: ∇^ ^ log det(−^) = ^ ^det The metavariable ^ > 0, which effectively controls the relative step size with respect to the barrier term in relation to the step size with respect to the goodness-of-fit function ^(^^), can be determined experimentally. It has been found that suitable values ​​for ^ lie in the range 0 < ^ ≤ 10^^. However, the requirement considered in the gradient descent method that det ^−^ ^^ ^^^^ , ^, ^^^ > 0 is not yet sufficient to ensure the matrix inequality ^ ^^ ^^^^ , ^, ^^ ≼ 0. Therefore, after each step of the gradient descent method, it must be checked whether the solution candidate found also satisfies the matrix inequality ^ ^^ ^^^^ , ^, ^^ ≼ 0. If this matrix inequality is not satisfied, the solution candidate must be rejected. According to the invention, the test to determine whether the matrix inequality is satisfied is carried out by Cholesky decomposition of the matrix −^ ^^ ^^^^ , ^, ^^.This is based on the finding that the Cholesky decomposition of an arbitrary matrix ^ exists if and only if ^ is symmetric and positive definite, that is, ^ = ^^ and ^ ≻ 0. To determine a starting value ^. ^ (for the matrix ^) as well as a starting value ^ ^ (for the parameter vector ^ of the policy KNN) for the start of the gradient method, it is suggested to use the starting value ^ ^ to be initialized randomly. Such a randomly initialized policy KNN typically exhibits an initial Lipschitz constant ^ ^,^^^^ on, which are below a predetermined limit ^ ^^^ lies and consequently a starting value ^ ^ can be determined. If the initial Lipschitz constant ^^,^^^^ of the randomly initialized policy KNN violates the condition ^^,^^^^ < ^^^^ , then the randomly initialized starting value ^^ of the parameter vector ^ of the policy KNN is projected onto a starting value ^^ that satisfies the condition. ^ ^^ ^^^^ , ^^, ^^^ ≼ 0 is satisfied, as explained above. In other words, a semidefinite optimization problem is solved that satisfies the constraint ^ ^^ ^^^^ , ^^, ^^ ≼ 0, and only the policy parameter vector ^ of the policy KNN remains as a decision variable, since the initial value ^^ for the matrix ^ is fixed. The constraint of such an optimization problem is (after assuming ^ = 0 and applying Schur's complements) linear and consequently convex in ^. With the cost function described above, the projection problem has a quadratic, but convex, cost function. For the sake of simplicity, the procedure is explained using only a simplified basic policy-critic approach; however, it can be transferred to any reinforcement learning procedure with an explicit policy. For such explicit policies, predetermined Lipschitz constants ^ can then be enforced.In a semidefinite restricted learning algorithm for fitting a policy KNN and a critical KNN, initial values ​​for the matrix ^ and for the parameter vector ^ of the policy KNN are determined, as described above. Furthermore, an initial value for the parameter vector ^ of the critical KNN and an initial value for the vector ^ of state parameters are determined, for example, by random initialization according to a predetermined probability distribution. Finally, learning rates (i.e., step sizes of the gradient descent method) ^^, ^^, ^^ are determined for fitting the parameter vector ^ of the policy KNN, the parameter vector ^ of the critical KNN, and the diagonal elements, respectively. The diagonal vector ^ of the matrix ^ is determined. Subsequently, a predetermined number of iteration steps are executed, which are subsequently indexed with ^. Each iteration step comprises a gradient step for parameter optimization and a policy rollout step. In a ^-th execution of the training iteration step, the following substeps are performed: ^ Based on the current policy and an exploration strategy (required for learning algorithms with deterministic policies) (i.e., determined by the output of the policy KNN), an action is determined based on the current state value. This action is parameterized by an action parameter vector ^^ ∼ ^^(^^|^^) (i.e., it is randomly selected according to a conditional probability distribution ^^(^^|^^)).For example, the Policy-KNN determines a (multidimensional) mean and a (multidimensional) standard deviation, which are used for the random generation of the action parameter vector ^. ^This can be used as a basis. In this way, the behavior of the Policy KNN and the Critic KNN can be explored for a certain local environment. Such exploration methods are known from the prior art. A successor state is determined using a Markov model ^^^^ ∼^(^^^^|^^ , ^^) (that is: randomly selected according to a conditional probability distribution ^(^^^^|^^ , ^^) which describes the environment on which the action stochastically selected by the Policy KNN, described by the action parameter vector ^^, acts). Explicit knowledge (a white-box model) of the conditional probability distribution ^(^^^^|^^ , ^^) is not strictly necessary. Rather, an experience tuple that describes the preceding state is sufficient for the learning process. ^ , the action parameter vector ^ ^, the subsequent state ^^^^ as well as the reward ^^.^ Update candidates are determined using a gradient step in each case:o to adjust the Critic-KNN based on a cost function ^^^^^, which represents the deviation of the evaluation determined by the Critic-KNN ^ ^ The goodness of an action (that is: the sum of discounted rewards expected in the future by the Critic-KNN) is captured by the current goodness of the action, which results from taking into account the experiences sampled (using experience tuples): ozur Adapting the Policy-KNN using a quality function ^(^^), which evaluates the action determined by the Policy-KNN with the help of the Critic-KNN, as well as using the barrier term already explained: To fit the diagonal vector ^ (and thus also the matrix ^) using the barrier term: ^^ ← ^ + ^ ^^ ∇^ ^log det ^−^^^ ^^^^ , ^, ^^^^^ It is checked whether the Cholesky decomposition of the matrix – ^^^ ^^^^ , ^^, ^^^ exists (that is, whether the matrix ^^^ ^ ^^^ , ^^, ^^^ is negative semidefinite). If this condition is met, the update candidates are adopted as new values: ^ ← ^ ^ ← ^^ ^ ← ^^If the condition is not met, the update candidates are rejected and a new iteration is started (that is, a new action and a new subsequent state are randomly determined). In a first aspect, the invention relates to a computer-implemented method for adapting a control device configured to detect a current state and determine an action based on the current state using a policy KNN and a critical KNN. The policy KNN has 1 + 1 layers connected via weight matrices , , … , where in a 1-th layer each ^ Neurons are arranged whose scalar activation function ^(^) in its rise between a minimum rise ^ and a maximum rise ^ according to are limited. The Policy KNN is configured to select an action based on a current state presented at its input, which can be parameterized via a current policy parameter vector ^ comprising the connection weights of the Policy KNN. The Critic KNN is configured to evaluate an action with an estimated reward value, which can be parameterized via a current Critic parameter vector ^ comprising the connection weights of the Critic KNN. In an initial step, an initial policy parameter vector ^ ^ and an initial critical parameter vector ^ ^ randomly selected. Furthermore, an initial state ^ ^ the environment is captured. In the initial step, an initial combination matrix is ​​created. ^ with diagonal structure and diagonal elements ^^^, which are summarized in a diagonal vector ^, initialized in such a way that for ^ = ^^ the symmetric Lipschitz matrix negative semidefinite.^ is formed from - a matrix ^, which, as block triangle matrices, comprises the weight matrices ^^ to ^^^^ from the first to the last hidden layer of the policy KNN, - a matrix ^, which is ribbon-like according to The unit matrices ^^^ , ^^^ , … ^^^ in the neuron counts ^^ , ^ = 1, … ^ from the first hidden layer to the output layer of the policy KNN are arranged in corresponding sizes and are formed with a predetermined positive Lipschitz constant ^^^^. In a subsequent iteration step, which is repeated several times, the following substeps are executed sequentially: - The policy KNN stochastically performs an action based on the current state presented at its input, according to an exploration strategy and the current policy parameter vector ^ ^ selected. - A subsequent state and a reward are selected based on the current state. ^ and the current promotion ^ ^An update candidate for the critical parameter vector ^ is determined by gradient descent with respect to a critical cost criterion, which captures the deviation of the reward value estimated by the critical KNN from a current reward value for the current action selected by the policy KNN. An update candidate for the policy parameter vector ^^ is determined by gradient ascent with respect to a restricted policy goodness criterion, which captures the sum of the estimated reward value and a barrier term log det −^ (^ ^^^^ , ^, ^ ). An update candidate for the diagonal vector ^^ is determined by gradient ascent with respect to the barrier term log det − ^ (^ ^^^^ , ^, ^ ). The Cholesky decomposition of the matrix −^ is examined. The update candidates ^, ^^, ^^ are then accepted as new parameters for processing the successor state if the Cholesky decomposition of the matrix - ^exists.Otherwise, the update candidates ^, ^^, ^^ are discarded. The method according to the invention enables the enforcement of desired robustness requirements of the control law and guarantees the achievement of these requirements. In other words, the parameter vector ^ of the policy KNN is optimized together with the symmetric positive semidefinite combination matrix ^ such that the Lipschitz matrix ^ (^ ^^^^ , ^, ^ ) remains negative semidefinite below a predetermined limit ^^^^ and, at the same time, the performance metric of the policy KNN is maximized under this constraint. In particular, robustness against input disturbances is achieved, which occur in technical applications, for example, due to measurement noise, actuator deviations, or actuator nonlinearities.In control devices designed or parameterized using conventional methods, such disturbances lead to arbitrary control interventions, increased energy consumption, excitation of higher-order system dynamics, or even destabilization of a control loop. Compared to alternative, heuristic methods for increasing robustness, improvements are achieved in terms of learning speed, reproducibility of the learning result, maximum achievable robustness, and / or control accuracy, all of which can be achieved for a defined robustness requirement. Furthermore, unlike conventional heuristic methods, a predetermined robustness can be guaranteed. In one embodiment of the method, an initial combination matrix ^ is generated in the initial step. ^ by determining that the initial policy parameter vector ^ ^ assigned initial Lipschitz constant ^^,^^^^ is determined. If the determined initial Lipschitz constant ^ ^,^^^^ greater than the predetermined positive Lipschitz constant ^ ^^^ is: ^^,^^^^ > ^^^^so by solving an optimization problem, the original initial policy parameter vector ^ ^ on a new initial policy parameter vector ^ ^ ^ The projected model is one for which the condition ^ ^^ ^^^^ , ^^, ^^^^ ≼ 0 with respect to the optimization problem is satisfied. This embodiment allows for a particularly fast initialization of the learning problem. In a further development of this embodiment, neurons are selected for the policy KNN that exhibit a minimal increase in the activation function from ^ = 0. The constraint of the projection problem, where the decision variables are the parameters of the policy KNN, is determined by applying Schur's complement according to simplified to a linear matrix inequality. By making the constraint in the decision variables linear, this further development enables particularly easy implementation and especially rapid initialization compared to random initialization of the learning method. In a second aspect, the invention relates to a computer-implemented method for evaluating the robustness of a control device that is configured to detect a current state and determine an action based on the current state using a policy KNN. The policy KNN has 1 + 1 layers connected via weight matrices 1, 1, ... 1, wherein neurons are arranged in a 1-th layer 1, 1 = 1, ... 1, whose activation function 1(1) has a slope between a minimum slope and a maximum slope according to are limited. The policy KNN is configured to select a current action based on a current state presented at its input and is parameterized via a current policy parameter vector ^ comprising the connection weights of the policy KNN. The robustness of the policy KNN is determined by a semidefinite programming method as the smallest Lipschitz constant ^ ^ ^ determines which one fulfills the criterion according to which the symmetric matrix is negative semidefinite. A matrix ^ here comprises, in a block diagonal structure, the weight matrices ^^ to ^^^^ of the policy KNN. A matrix ^ here comprises according to Ribbon-like arranged unit matrices ^^^ , ^^^ , … ^^^. A combination matrix ^ with a diagonal structure comprises arbitrarily selectable non-negative diagonal elements in semidefinite programming. ^^The robustness determined by this method enables a particularly reliable estimation of the effects of input disturbances on the outputs of the policy KNN. Further advantages correspond to the advantages of the method according to the first aspect of the invention. In a third aspect, the invention relates to a control device configured to detect a current state and determine an action based on that current state. According to the invention, the control device comprises a policy KNN that has been trained using a method according to the first aspect of the invention and / or whose robustness has been evaluated using a method according to the second aspect of the invention. Based on a current state, the policy KNN determines the action according to its parameterization with the policy parameter vector ^.The advantages of the control device according to the invention correspond to the advantages of the method according to the first aspect of the invention and the advantages of the method according to the second aspect of the invention. The control device according to the invention can be used to control a technically selectable function of a technically arbitrary system. By way of example, the control device can be configured as a controller that acts on a controlled system, which can be considered the environment. Furthermore, the control device can be configured as a predictor for estimating a future system state of a technical system. Based on such an estimate, actions can be determined to bring about a desired future system state or to avoid an undesired future system state.In one embodiment, the control device is configured to control a vehicle function. For example, the control device for steering is configured such that a desired (predefined) trajectory of the vehicle is followed with particularly small steering errors. The control device can additionally or alternatively be configured to control an engine function, a braking function, cruise control, lane keeping, lane changing, or other vehicle functions, or combinations of such vehicle functions. This enables particularly safe and reliable vehicle operation, meaning that deviations from desired behavior are especially small and / or highly improbable.Furthermore, this enables particularly efficient vehicle operation, for example, particularly low fuel or energy consumption and / or particularly low wear on vehicle components such as the engine and steering system. It also enables particularly comfortable vehicle operation, for example, by making interventions in vehicle functions (steering movements, acceleration or braking, or similar interventions) particularly infrequent and / or minimal. Exemplary embodiments of the invention are explained in more detail below with reference to a drawing. Figure 1 shows a flowchart for a reinforcement learning method for adapting a policy KNN and a critical KNN. In the flowchart shown in Figure 1, an initialization is performed in a first step S1 (initialization step), starting from a starting point S0.During initialization, the weights of the Policy KNN (summarized in the Policy parameter vector ^^) and the weights of the Critic KNN (summarized in the Critic parameter vector ^) are used. ^The matrix ^ is randomly initialized. Furthermore, learning rates (i.e., step sizes) ^^, ^^, ^^ are determined for the gradient methods to fit the parameter vector ^ of the policy KNN, the parameter vector ^ of the critical KNN, and the diagonal elements ^^^ of the diagonal vector ^ of the matrix ^, respectively. Additionally, the diagonal elements ^^^ of the matrix ^ are initialized such that the condition ^ ^^ ^^^^ , ^, ^^ ≼ 0 is satisfied. The initialization of the matrix ^ can, for example, be repeated randomly until the condition ^ ^^ ^^^^ , ^, ^^ ≼ 0 is satisfied. Alternatively, as already described, an initial assignment of the matrix ^ can also be found by solving a linear optimization problem and projecting the randomly initialized parameter vector ^^ of the policy KNN onto a modified starting value ^^^, which satisfies the constraint (given an unchanged starting value for the matrix ^).Subsequently, a second step S2 (iteration step) is repeated for a predetermined number of iterations. The second step S2 comprises substeps TS1 to TS4. In the first substep TS1, an action is randomly selected based on the current policy (i.e., determined by the output of the policy KNN), which is parameterized by an action parameter vector ^^ ∼ ^^(^^|^^) (i.e., an action ^^ is randomly selected according to a conditional probability distribution ^^(^^|^^) determined by the output of the policy KNN). In the subsequent second substep TS2, the selected action is applied to the environment. A subsequent state ^^^^ is determined through interaction with the environment. A reward value is determined based on the subsequent state. In the subsequent third substep TS3, update candidates are identified. For the critical parameter vector ^, an update candidate ^ is determined by applying the associated learning rate ^.^ a gradient descent along the cost function is carried out, which records the deviation between the Critic-KNN's assessment of the action and the current assessment of the action: For the policy parameter vector ^, an update candidate ^^ is determined by applying the associated learning rate ^. ^ a gradient ascent along the quality function ^ ( ^ ^ ) , which evaluates the action determined by the policy KNN, and is carried out along the barrier term: For the diagonal vector ^ of the combination coefficients ^^^ , ^ = 1 … ^ on the diagonal of the matrix ^, an update candidate ^^ is determined by applying the assigned learning rate ^ ^ A gradient ascent is performed along the barrier term: ^^ ← ^ + ^ ^^ ∇^ ^log det ^^^ ^^^^ , ^, ^^^ In a subsequent fourth substep TS4, the matrix updated using the update candidates ^^, ^, ^^, is determined and checked whether the Cholesky decomposition of −^^^ ^ ^^^ , ^, ^^ exists. (In other words: it is checked whether ^^^ ^^^^ , ^, ^^ is a negative semidefinite). If this condition is met, the update candidates are accepted in a subsequent fifth substep TS5. That is, the assignments are ^ ← ^^ If the condition that ^^^ ^^^^ , ^, ^^ is negative semidefinite is not met, the update candidates are rejected in a sixth substep TS6 as an alternative to the fifth substep TS5. That is, the policy parameter vector ^, the critical parameter vector ^, and the diagonal elements The values ​​of the matrix ^ remain unchanged. The second step S2 is repeated until a predetermined number of iterations is reached. Alternatively, other termination criteria are possible, for example, by comparing the performance function ^(^^) with a threshold value.

Claims

Mercedes-Benz Group AG Patent Claims 1. Computer-implemented method for adapting a control device which is configured to detect a current state and to determine an action based on the current state by means of a policy artificial neural network (policy KNN) and by means of a critical artificial neural network (critical KNN), wherein the policy KNN has 1+1 layers connected via weight matrices, wherein in a 1-th layer 1 neurons are arranged whose scalar activation function 1(1) has a slope between a minimum slope and a maximum slope according to are restricted, - the Policy-KNN is configured to select an action based on a current state presented at its input, which can be parameterized via a current policy parameter vector ^ comprising the connection weights of the Policy-KNN, - the Critic-KNN is configured to evaluate an action with an estimated reward value, which can be parameterized via a current Critic parameter vector ^ comprising the connection weights of the Critic-KNN, characterized in that - in an initial step (S1) an initial Policy parameter vector ^^ and an initial Critic parameter vector ^^ are randomly selected as well as an initial state ^ ^ the environment is captured, -in the initial step (S1) an initial combination matrix ^^ with diagonal structure and diagonal elements ^ ^^, which are summarized in a diagonal vector ^, is initialized such that for ^ = ^^ the symmetric Lipschitz matrix negative is semidefinite, where ^ is a matrix ^, which, as block triangle matrices, are the weight matrices^ ^ until ^ ^^^ from the first to the last hidden layer of the policy- K NNs umfasst, o a matrix ^, which is ribbon-like according to arranged unit matrices ^^^ , ^^^ , … ^^^ in the neuron counts^^ , ^ = 1, … ^ from the first hidden layer to the output layer of the policy KNN comprises sizes corresponding to a predetermined positive Lipschitz constant ^^^^, - in a subsequently repeated iteration step (S2)o, the policy KNN stochastically performs an action based on the current state presented at its input according to the current policy parameter vector ^, according to an exploration strategy. ^is selected, a subsequent state and a reward are determined based on the current state and the current action, an update candidate for the Critic parameter vector is determined by gradient descent with respect to a Critic cost criterion, which captures the deviation of the reward value estimated by the Critic KNN from a current reward value for the current action selected by the Policy KNN, an update candidate for the Policy parameter vector is determined by gradient ascent with respect to a restricted Policy quality criterion, which captures the sum of the estimated reward value and a barrier term log det −^ (^ ^^^^ , ^, ^ ), an update candidate for the diagonal vector is determined by gradient ascent with respect to the barrier term log det ^ (^ ^^^^ , ^, ^ ), the Cholesky decomposition of the matrix −^ is examined, and the Update candidates ^, ^^,^^ then be accepted as new parameters for processing the subsequent state if the Cholesky decomposition of the matrix exists, and are otherwise discarded.

2. Method according to claim 1, characterized in that - in the initial step (S1) an initial combination matrix ^^ is determined by applying a matrix corresponding to the initial policy parameter vector ^, ^ The assigned initial Lipschitz constant ^^,^^^^ is determined, and—if ^^,^^^^ > ^^^^—the original initial policy parameter vector ^ is found by solving an optimization problem. ^ is projected onto a new initial policy parameter vector ^^^ for which the condition ^ ^^ ^^^^ , ^^, ^^^^ ≼ 0 is satisfied.

3. Method according to claim 2, characterized in that - for the policy KNN neurons are selected which have a minimal slope of the activation function from ^ = 0, and - the constraint of the projection problem, in which the decision variables are the parameters of the policy KNN, is satisfied by applying Schur's complement according to simplified to a linear matrix inequality.

4. Computer-implemented method for evaluating the robustness of a control device configured to detect a current state and determine an action based on the current state using a policy KNN, wherein the policy KNN has 1 layers connected via weight matrices, 1, 2, ... 3, where neurons are arranged in a 1-th layer, 1, ... 4, whose activation function has a slope between a minimum slope and a maximum slope. according to are restricted, -the policy KNN is set up to select a current action based on a current state presented at its input and is parameterized via a current policy parameter vector ^ comprising the connection weights of the policy KNN, characterized in that -the robustness is determined by a semidefinite programming method as the smallest Lipschitz constant ^ ^ ^ is determined which fulfills the criterion according to which the symmetric matrix negative semidefinite, where - a matrix ^ in block diagonal structure comprises the weight matrices ^^ to ^^^^ of the policy KNN, - a matrix ^ band-like according to 5. Control device configured for detecting a current state and determining an action based on the current state, characterized in that the control device comprises a policy KNN trained with a method according to one of the preceding claims.

6. Control device according to claim 5, characterized in that The control device is set up to control a vehicle function of a vehicle.

Images