Method for training a control strategy for a technical system

Abstract

According to various embodiments, a method for training a control strategy for a technical system is provided, comprising controlling the technical system and determining a sequence of states of the technical system that occur during the control of the technical system, estimating the transition dynamics between successive states of the sequence of states, detecting changes in the transition dynamics during the sequence of states based on the estimates of the transition dynamics, and, in response to the detection of a change in the transition dynamics from a first transition dynamic to a second transition dynamic, switching the decision rule according to which the technical system is controlled and which is trained on the sequence of states, from a first decision rule associated with the first transition dynamic to a second decision rule associated with the second transition dynamic.

Description

[0001] The present disclosure relates to methods for training a control strategy for a technical system.

[0002] Reinforcement learning is an effective method for learning control strategies for technical systems, such as robots or vehicles. However, it is typically based on the assumption that the transition dynamics, i.e., the rules according to which state transitions of the controlled system (and possibly its environment, such as objects manipulated by a robot arm) occur in response to selected control actions, remain (at least essentially) the same throughout the control process. However, this assumption is not justified for certain control systems, such as the ABS (anti-lock braking system) control of a vehicle, where the transition dynamics can change abruptly and significantly, since, for example, wheels lock much more easily on a dirt road than on asphalt.Therefore, approaches are desirable that enable the training of a control strategy (and effective control) in applications where the transition dynamics can change abruptly and significantly.

[0003] According to various embodiments, a method for training a control strategy for a technical system is provided, comprising controlling the technical system and determining a sequence of states of the technical system that occur during the control of the technical system, estimating the transition dynamics between successive states of the sequence of states (ieEstimating a transition dynamics for a plurality of state transitions (between successive states in the sequence of states), detecting changes in the transition dynamics over the course of the sequence of states based on the estimates of the transition dynamics and, in response to the detection of a change in the transition dynamics from a first transition dynamics to a second transition dynamics, changing the decision rule (or "partial control strategy", where the control strategy comprises a plurality of such partial control strategies) according to which the technical system is controlled and which is trained based on the sequence of states, from a first decision rule associated with the first transition dynamics to a second decision rule associated with the second transition dynamics.

[0004] The procedure described above enables the training of a control strategy that is able to detect and adapt to changes in the transition dynamics, ie, to switch to a new decision rule (or to switch to an earlier decision rule if the detected transition dynamics already existed at a previous time).

[0005] This is realized, for example, by means of three components: an identifier for the dynamics (e.g., using a detection of changes in the dynamics using a cumulative sum (“CUSUM”)), an RL (Reinforcement Learning) agent (e.g., learning using Soft Actor Critic), and a memory for properties of previous environments (e.g., environmental dynamics).

[0006] According to various embodiments, the procedure described above can be seen as the training or subsequent use based on a piecewise Bayes-adaptive decision process and is therefore also referred to below as PS-BAMDP method (for Piece-Wise Bayes Adaptive Markov Decision Process). The control strategy (e.g., a corresponding control device, such as an ABS controller) contains a component that, during its interaction with the environment (the properties of which can change abruptly, such as when changing lanes), observes the properties of the environment and attempts to detect whether there are changes in these properties, in particular the transition dynamics (i.e., in the way states of the controlled technical system and / or the environment) change depending on control actions. Since the input regarding the states from the state space can be continuous (e.g.,vehicle speed values ​​can occur in a continuous range), this component clusters the states (for a specific time window) and creates a transition matrix (for the time window) based on the clustering. If it detects changes in the transition matrix (from time window to time window), it indicates (e.g., triggers a signal) that a change in the transition dynamics has occurred. Furthermore, based on the transition matrix (after a change), it tries to understand whether a transition dynamic represented by such (or at least a similar) transition matrix has already occurred in the past (i.e., the control strategy has already (at least partially) learned a decision rule for this transition dynamic). If so, it calls the decision rule for this transition dynamic and continues RL learning for it (or uses it for control if it is already fully trained).If not, a new decision rule is initialized and trained to deal with this new type of transition dynamics.

[0007] This enables the training of a control strategy that can handle several different transition dynamics, while retrieving decision rules for those transition dynamics for which it has already been trained. This eliminates the need for a large (contextual) decision rule (whose context is derived from the detected transition dynamics) to handle the different transition dynamics. Furthermore, this approach provides a certain degree of explainability, since the decision rule used reveals which transition dynamic the control strategy (or a particular control device) currently believes it is interacting with. A decision rule can be understood as a mapping from state to control action (although this mapping may include a certain degree of randomness for the purpose of exploration).

[0008] The approach described above can be used for any control system where the control device does not know in advance what transient dynamics it will encounter. ABS control is a good example, as an ABS system is typically tuned to specific, limited scenarios defined during its manufacture. The approach described above, however, allows it to adapt to new scenarios as it interacts with them.

[0009] Various examples of implementation are given below.

[0010] Embodiment 1 is a method for training a control strategy for a technical system as described above.

[0011] Embodiment 2 is a method according to embodiment 1, comprising, in response to detecting a further change in the transition dynamics, switching back to the first decision rule in response to a determination that the transition dynamics have changed in the further change to the first transition dynamics.

[0012] This ensures that when the original conditions are restored—for example, a vehicle first drives on asphalt, then on a dirt road, and then back on asphalt—good control results can be achieved immediately on the asphalt, as this has already been trained. For example, it is determined that the transition dynamics have changed during the further change compared to the first transition dynamics if the transition dynamics after the further change differ from the first transition dynamics by less than a specified threshold (in terms of a specified metric).

[0013] For example, a set of decision rules can be defined, where each decision rule is associated with a respective transition dynamic, and where the first decision rule is included in the set and the second decision rule is added to the set of decision rules if it is not already included in the set of decision rules.

[0014] Embodiment 3 is a method according to embodiment 1 or 2, wherein changing the decision rule to the second decision rule is a resetting of the decision rule to a predetermined initial version.

[0015] For example, the second decision rule is a reverted version of the first decision rule, from which the first decision rule was trained. In other words, if it is determined that the transition dynamics have changed and thus the training as performed for the first decision rule is presumably inadequate, it is possible to switch back to an initial version of the decision rule.

[0016] Embodiment 4 is a method according to one of embodiments 1 to 3, comprising training the first decision rule based on states and state transitions of the sequence of states before the change, wherein the training of the first decision rule comprises learning a first transition dynamics model for the first transition dynamics and the first decision rule is trained using the first transition dynamics model, and / or comprising training the second decision rule based on states and state transitions of the sequence of states after the change, wherein the training of the second decision rule comprises training the second decision rule, learning a second transition dynamics model for the second transition dynamics and the second decision rule is trained using the second transition dynamics model.

[0017] For example, a critic (for the first and second decision rules, respectively) can use the environmental dynamics model to evaluate the actions chosen by an actor (for the first and second decision rules, respectively). The first environmental dynamics model and the second environmental dynamics model can be fitted to the respective state transitions according to Bayesian inference (see discussion below in the context of BAMDPs).

[0018] Embodiment 5 is a method according to one of the embodiments 1 to 4, wherein the transition dynamics are estimated by encoding states of a subsequence of the sequence of states (possibly together with associated actions and / or rewards, i.e. the encoding and thus also the estimation of the transition dynamics can depend on rewards and actions that bring about the state transitions) into a sequence of state representations in a latent space by means of a (e.g. variational) autoencoder, and the transition dynamics for the states of the subsequence are estimated in the form of a transition kernel between elements of the latent space.

[0019] The use of coding reduces the complexity of change detection. The autoencoder used for coding can be part of the respective decision rule (i.e., the decision rule currently being controlled, i.e., the first decision rule for states before the change). For example, a planning module of the respective decision rule can select an action based on the coding of a respective state. The autoencoder can be trained together with the respective decision rule and also reset if necessary.

[0020] Embodiment 6 is a method according to embodiment 5, wherein each state representation of the sequence of state representations is assigned to a cluster and transition dynamics in the form of a transition kernel between elements of the latent space representing the clusters to which the state representations are assigned are estimated.

[0021] The use of clusters enables robust change detection.

[0022] Embodiment 7 is a method according to embodiment 6, wherein a first transition kernel is estimated for a first subsequence of the sequence of states (as described above for the subsequence) and for a second subsequence of the sequence of states (as described above for the subsequence) that follows the first subsequence, and a change in the transition dynamics is detected based on a ratio of the second transition kernel to the first transition kernel (or difference between the two).

[0023] The first subsequence and the second subsequence, corresponding to different time windows, for example, and the transition kernel are estimated for each of them, and the change is detected using changepoint detection. This enables reliable detection of changes.

[0024] Embodiment 8 is a method according to one of the embodiments 1 to 7, wherein the first decision rule and the second decision rule (and possibly further decision rules of the control strategy) are trained by means of reinforcement learning.

[0025] The use of reinforcement learning enables effective training of the control strategy (i.e., its various decision rules), especially in the context of the approach described above, where for each decision rule, the transition dynamics can be assumed to be constant, since when it changes, the decision rule is changed.

[0026] Embodiment 9 is a method for controlling a technical system, comprising: training a control strategy with a plurality of decision rules, each associated with a transition dynamic, according to one of embodiments 1 to 8 and controlling the technical system according to the trained control strategy, wherein changes in the transition dynamics are detected and, in response to the detection of a change in the transition dynamics, the decision rule according to which control is carried out is changed to a decision rule of the control strategy that is associated with the transition dynamics to which the transition dynamics changes.

[0027] Embodiment 10 is a method according to one of the embodiments 1 to 9, wherein the technical system is the brake of a vehicle.

[0028] For example, the control is an ABS control.

[0029] Embodiment 11 is a control device (e.g. vehicle dynamics controller) which is configured to carry out a method according to one of the embodiments 1 to 10.

[0030] Embodiment 12 is a computer program including instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 10.

[0031] Embodiment 13 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 10.

[0032] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Fig. 1 shows a vehicle. Fig. 2 illustrates the training of a control strategy. Fig. 3 shows a flowchart illustrating a method for training a control strategy for a technical system according to an embodiment.

[0033] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0034] Various examples are described in more detail below.

[0035] Fig. 1 shows a vehicle 101.

[0036] In the example of Fig. 1, a vehicle 101, for example a car or truck, is provided with a vehicle control device 102.

[0037] The vehicle control device 102 has data processing components, e.g., a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software according to which the vehicle control device 102 operates and data processed by the processor 103.

[0038] For example, the stored control software (computer program) includes instructions that, when executed by the processor, cause the processor 103 to implement a machine learning model 105.

[0039] The vehicle control device 102 can perform various control tasks ranging from driver assistance functions to fully autonomous driving. The vehicle control device 102 determines the respective control actions using the machine learning model 105 (i.e., the learning model 105 implements a control strategy).

[0040] For example, according to one embodiment, the vehicle control device 102 is a control device for an anti-lock braking system (ABS), ie, generates a desired braking pressure (also referred to as input (wheel) braking pressure) for a brake 106.

[0041] The vehicle control device 102 (or the control strategy implemented by it) is trained, for example, using reinforcement learning (RL). Training the vehicle control device 102 can consist of training the machine learning model 105 on an external device (training device or training (computer) system) and then loading it onto the vehicle control device 102, e.g., using simulations (alternatively, the vehicle 101 itself can be implemented as a simulated vehicle in a simulation environment, thus training the vehicle control device 102).

[0042] In reinforcement learning, an attempt is made to train an agent (or a control strategy that the agent follows), in the above example the vehicle control device 102 (or the machine learning model 105) in such a way that a certain reward is as large as possible (in the above example, for example, depending on how successfully braking was carried out).

[0043] Many reinforcement learning methods typically assume that the environment is stationary. However, in many realistic cases, the dynamics of the environment or the reward for the agent may change abruptly during a trajectory (i.e., a sequence of states, possibly with corresponding rewards and actions), requiring rapid adaptation of the control strategy to ensure safety and performance. This is especially true when the set of possible environments is continuous, as the agent can then no longer accurately identify the true environment.

[0044] For example, in the case of the ABS control mentioned above, the control device 102 performs braking in different ways depending on the road on which the vehicle 101 is traveling. For example, it may be a concrete road, a sand road, etc. The dynamics of each road are quite different with respect to the ABS system, and the transition between the different road types is abrupt (e.g., the driver turns from a concrete road to a sand road).

[0045] For example, to ensure optimal safety and decide when to activate ABS, the agent (i.e., controller 102 in the above example) must be precisely tuned to the tire dynamics and the road-tire interactions. However, both depend on the type of road and weather conditions, two characteristics that can change over time.

[0046] In the above control scenario, the environment is determined by non-stationary, (from the agent's perspective) unobserved parameters. It can be modeled as a partially observable Markov decision process (POMDP), where the transition function (i.e., the transition dynamics) and the reward function represent the unobserved state. However, in contrast to the general POMDP, where the hidden state changes at each step, in the above control scenario, the hidden observations vary at irregular time intervals (e.g., when changing lanes). Instead of constantly maintaining an assumption about hidden states, various embodiments exploit this additional structure (change points at different time intervals) and derive an MDP model between the individual change points.

[0047] A non-stationary Markov decision process (MDP) is a tuple (S, A, γ, p, r), where S and A are the state and action spaces, respectively, and γ ∈ [0, 1] is a discount factor. Each element of the state space represents a state of the respective controlled technical system (which may include state information about the environment of a controlled device, e.g., when a robot manipulates objects). In the above example of ABS control, this represents a state of the vehicle (speed, information about whether the wheels are locked, etc.). Actions from the action space, for example, are different braking forces.

[0048] The transition parameter p := (p t (·|s, a)) t∈ℕ is a collection of transition kernels, ie, p t (|s, a) ∈ Δ S represents a probability over S for all t ∈ ℕ, s ∈ S, a ∈ A. The reward parameter r := (rt(s, a)) t ∈ ℕis a collection of real-valued functions (s, a) ↦ rt(s, a) ∈ ℝ that model the immediate performance of the agent. At time t ∈ ℕ, the agent and its environment are in state s t and he carries out the action a t according to a decision rule π t : S → Δ A (ie maps each state to a probability distribution over the actions; here the control strategy contains different decision rules π t for different points in time, e.g. depending on which lane the vehicle 101 is currently moving on). The collection of decision rules π := (π t ) t∈ ℕ defines the control strategy.

[0049] The control strategy is trained with the goal of maximizing the expected discounted sum of rewards υpπ(s):=Epπ[∑t=0∞γtrt(st,at)|s0=s]

[0050] This quantity is the value function of the non-stationary MDP.

[0051] In general, p and r are unknown, and decisions must be made over an infinite time horizon. This is an extension of the model-based finite-time-horizon case, where dynamic programming can account for non-stationary model parameters.

[0052] Furthermore, the time dependence of p and r prevents the control strategy from being optimized using standard Bellman operators, as these are based on stationarity. If (p t , r t ) =: (p, r) is constant for all t ∈ ℕ, the MDP is stationary and is denoted by (S,A, γ, p, r). Similarly, a control strategy π is stationary if π t =: π for all t ∈ ℕ. It is then simply denoted by π. A consideration of such control strategies is sufficient for stationary MDPs.

[0053] In complex environments, the transition function and the reward function can be fully represented by abstraction with (relatively) few parameters. Let θ := (θ t ) t∈ℕ , where, for all t ∈ ℕ, θ t belongs to a parameter space Θ such that (p θ , r θ ) := (p θt , r θt ) t ∈ ℕ is an approximation of (p, r). According to various embodiments, such an approximation is used for a representation of an MDP model by a neural network (θ then represents, for example, the weights of the neural network, or of several neural networks, namely one for each transition dynamic (and possibly also each reward function), indexed by the t). According to various embodiments, a parametric control strategy of the form π ψ := (π ψt ) t∈ℕconsidered (where ψ is, for example, a neural network that implements the control strategy (or several neural networks for different transition dynamics)).

[0054] Since an accurate determination of the true environmental conditions (especially the transient dynamics) may be impossible, especially when the parameter space is continuous (as in ABS control), a Bayesian reinforcement learning (RL) approach is used according to various embodiments.

[0055] In a Bayesian adaptive MDP (BAMDP), the agent has a probability distribution over possible environmental conditions and uses newly acquired information to reduce the uncertainty about the environmental conditions.

[0056] For this purpose, it is assumed that the transition function and the reward functions follow an a priori distribution b(·) over possible (transition and reward) models (p, r), and this assumption (i.e., the distribution over the possible models) is updated based on newly acquired experience. Let τ :t := (s0, a0, r1, s1, a1, ···, r t , s t ) is a trajectory that the agent has experienced up to time t. The agent then updates its belief using Bayes' rule, i.e. b(·|τ :t ) ∝ ℙ(τ :t |·)ℙ(·). Thus, uncertainty regarding the transition dynamics (and rewards) can be accounted for by having the agent make decisions based on the current assumption about the transition dynamics (and rewards).

[0057] Given a stationary MDP (S, A, γ, p, r), a BAMDP is given by (S + , A, γ, p + , r + ), where S +:= S × B and B is a set of distributions (for the possible models (p, r)). The extended (transition and reward) model (p + , r + ) is defined as: p+((s',b')|(s,b),a):=1{b'=b(⋅|τ:t+1)}E(p,r)∼b[p(s'|sa)], r+((s,b).a):=r(sa), for all hyperstates (s, b), (s', b') ∈ S+ and a ∈ A. In this case, a function π defines + : S + → Δ A a Bayesian adaptive control strategy trained with the goal of maximizing its value function: υp+π+:=Ep+π+[∑t=0∞γtr+((st,bt),at)]

[0058] Theoretically, BAMDPs can be solved using Bellman equations in a hyperstate space. The variBAD algorithm uses variational inference to scale a Bayesian adaptive control strategy to large problems. An amortized inference network (encoder) approximates q ϕ (·|τ :t ) the belief b(·|τ :t), so that the encoder q ϕ can be used to simulate new trajectories in the resulting BAMDP and to derive the reconstruction loss, i.e., the objective to be minimized by the associated decoder. In addition, variBAD trains a strategy π ψ (a|s, qϕ(·|τ :t )), which is based on the approximate assumption to maximize return.

[0059] Although Bayesian-optimal strategies solve a single unknown task with low sample complexity, they do not consider the temporal evolution of the dynamics. Indeed, the BAMDP formalism is based on a stationary but unknown MDP. Therefore, extending the original state space by the belief is not about coping with model changes, but rather about quickly achieving good performance despite model uncertainty.

[0060] To enable control in environments whose properties can change (so that, in particular, the transition dynamics change), according to various embodiments, a control strategy based on piecewise stationary Bayes-adaptive Markov decision processes (PS-BAMDPs) is trained. A PS-BAMDP is a sequence of stationary BAMDPs and can be seen as an extension of BAMDPs to time-dependent MDPs. For each BAMDP of the sequence, the control strategy can be trained individually (e.g., as described above using variBAD). The stationary BAMDPs that change the PS-BAMDP differ in their latent parameters, i.e., over the course of the PS-BAMDP, the latent parameters evolve according to a renewal process. In the ABS control application case, for example, one BAMDP corresponds to an asphalt road and another to a dirt road, i.e.The latent parameters represent the different road surface properties that the control system cannot directly observe.

[0061] According to various embodiments, the goal of training (here, reinforcement learning) is thus to train a non-stationary Bayesian optimal control strategy that creates an optimal balance between exploration and exploitation and adapts quickly to abrupt changes (especially transition dynamics). For this purpose, according to various embodiments, an algorithm known, for example, as a Bayesian Adaptive Quickest Change Detector (BAQCD) is used. This algorithm detects changes in the environment by detecting change points and learns Bayesian optimal strategies accordingly. In addition to managing the continuity of the parameter space, this achieves high performance and adaptability.

[0062] In the following, PS-BAMDPs are described in more detail. First, a non-stationary BAMDP (NS-MDP) is defined: Let (S,A, γ, p, r) be a non-stationary MDP. A non-stationary BAMDP (NS-BAMDP) is a tuple (S + , A, γ, p + , r + ), where p + := (p t + ) t ∈ N, r + := (r t + ) t∈ℕ and where for all t ∈ N (S + , A, γ, p t + , r t + ) is a BAMDP. For an NS-BAMDP, a control strategy is represented by a sequence of adaptive Bayesian control strategies π + := (π t + ) t ∈ ℕ given the υp+π+:=Ep+π+[∑t=0∞γtrt+((st,bt),at)] should maximize.

[0063] Similar to non-stationary MDPs, there is no systematic method for optimizing this objective function. In practice, however, environmental parameters may remain constant for an (unknown) period of time until they undergo a change (as in ABS control, where the vehicle remains on the same roadway for a while). Therefore, if environmental parameters change infrequently enough, one can use the data (observations, e.g., braking behavior for a specific roadway) between the change times and train a respective decision rule (i.e., a decision rule for the respective transition dynamics) within such a time interval (between two change times).

[0064] For this purpose, a piecewise BAMDP (PS-BAMDP) is defined as follows: Let (λ k ) k ∈ ℕ an increasing sequence of positive integers. A piecewise stationary BAMDP (PS-BAMDP) is an NS-BAMDP (S + , A, γ, p+ , r + ), for which for all t ∈ ℕ holds: pt+=∑k=0∞1[λk,λk+1)(t)pλk+, rt+=∑k=0∞1[λk,λk+1)(t)rλk+.

[0065] The λ k are the change points, i.e., the points in time at which the transition dynamics change (so significantly that a different decision rule is used). The change points are generally unknown and are therefore derived from state samples.

[0066] This can be done with change point detection (CDP) or Quickest Change Detection QCD as follows: Let (X t ) t ∈ ℕ a sequence of samples (X t ) t ∈ ℕ , such that for all t ∈ ℕ, X t according to f ϕt (·|X t-1 ), where ϕ t∈ Φ is the underlying (transition dynamics) parameter. Furthermore, let k ∈ ℕ be a counting index (as in the PS-BAMDP definition above). In CPD (or QCD) it is assumed that ϕ t := ϕ k up to an unknown step λ k ∈ ℕ remains constant and then changes. From this step λ k on remains ϕ t = ϕ k+1 to the next change point λ k+1 > λ k etc. In other words: For all t ∈ ℕ, X t+1 ∑k=0∞fϕk(⋅|Xt)1[λk,λk+1)(t+1)

[0067] The goal of CPD is to find all change points (λ k ) k ∈ ℕ quickly. In practice, it is a matter of finding a suitable stochastic process S := (S t ) t ∈ ℕ to design a stop time of the form τk+1S:=min{t∈ℕ:t>τk.St>h} indicates a change point. The process S is called the statistic, and the parameter h is called the detection threshold. When developing an appropriate statistic, the delay in detection must be traded off against the frequency of false alarms. Typically, the CUSUM statistic and the Shiryaev statistic are used because they are optimal under certain conditions. The CUSUM statistic is defined as CUSUM(S, x) := (S + x)+, and the Shiryaev statistic as SHIRYAEV(S, x) := ex(1 + S).

[0068] Regarding the detection of changes in PS-BAMDPs, in the Bayesian framework, a change point is a random variable with values ​​in N with a geometric prior distribution. According to various embodiments, (λ k ) k∈ℕ considered as a renewal process, ie for all k∈ℕ the change point increments λ k+1 - λ kindependently and identically distributed according to a geometric distribution of the parameter p. Thus, the probability that a change occurs is zero at each change point and increases with time until the next change. This fits particularly well in the context of ABS systems, as it means that the control conditions remain stable for a certain period of time, while the probability that they change increases over time.

[0069] The following describes how changes are detected in PS-BAMDs. Between two changes, the PS-BAMD is a BAMD, meaning that between two changes, as described above, there is a belief about the environment (i.e., the current control conditions), namely a distribution b(.) over the possible transition dynamics (or, more generally, environmental models), which is adjusted using Bayes' rule (between two change points). A change is a change in the distribution b. k (.) to another distribution b k+1 (.), which in turn is adjusted using Bayes' rule.

[0070] For this purpose, according to various embodiments (similar to variBAD), a latent representation of the assumption about the environment is used, as described below with reference to Fig. 2 is described.

[0071] Fig. 2 illustrates the training of a control strategy.

[0072] As in Fig.1, a currently observed trajectory τ :t (e.g., the speed curve of the vehicle and / or its wheels for a currently performed braking operation) is first input into the encoder 201 of a variation autoencoder (VAE). The function implemented by the encoder 201 is represented by q ϕ Its output is a latent representation (µ :t , σ :t ), i.e. for each state of the trajectory (which has a sequence of states and associated rewards) a sequence of specifications of probability distributions (here normal distributions) in a latent space 202, where each probability distribution specification has the form of a respective pair of mean µ and standard deviation σ.

[0073] The latent representation of a trajectory (which in particular contains a sequence of states) is therefore a sequence of latent individual representations (µ, σ), with one (“state”) representation per state (i.e. time index) of the trajectory.

[0074] The VAE is trained so that its decoder 203 can extract from the latent representations of a trajectory τ :t a reconstruction τ̂ :t which corresponds as closely as possible to the trajectory τ :t matches.

[0075] The decoder 203 takes as input encoded states of a trajectory in the latent space and decodes them to reconstruct parts of the trajectory. For example, the VariBAD algorithm is used to, given τ :t the next state s t+1 and the next reward r t+1to reconstruct. The idea can be seen in the fact that if the encoded states make it possible to predict the next transition and reward, all relevant information about the environment has been encoded.

[0076] In the latent space 202, a clustering 204 is performed, ie latent state representations are grouped into clusters and each cluster is represented by a respective state representative, so that a trajectory can be created by assigning the state representations to the clusters by a sequence of state representatives ĉ :t = (µ̂ :t , σ̂ :t ) is represented.

[0077] Then an estimate 205 of the transition dynamics is made (e.g. in the form of a transition kernel or a transition matrix) P(ĉ i+1 |ĉ i ).

[0078] Then, a change point detection 206 is performed (e.g., using the CUSUM statistic in the Markov version) to determine whether the transition dynamics have changed compared to a previous transition dynamics (ie, compared to a transition dynamics determined for a previously observed and similarly processed trajectory).

[0079] The data encoded (by the encoder 201 of the VAE) are clustered, which creates a “predicted” trajectory ĉ for a current trajectory. :t in the latent space 202. This allows the estimation of a transition kernel from one cluster to another using maximum likelihood estimation: Let T0 and T1 be the sizes of two consecutive, non-overlapping time windows W0 and W1 with the indices: W0:=(t−T0−T1,...,t−T1−1) W1:=(t−T1,...,t−1)

[0080] It should be noted that the windows, especially the window W0, can extend from a current cluster into another cluster (than the one to which the state representations of the current trajectory belong) if the time index goes back far enough. In the transition dynamics P(ĉ i+1 ,|ĉ i ) can therefore ĉ i+1 and ĉ i belong to different clusters (this can also be the case without there being a jump in the transition dynamics).

[0081] Based on these windows, a transition kernel P0(ĉ i+1 |ĉ i ) for i from the window W0 and a transition kernel P(ĉ i+1 |ĉ i ) for i from the second window. Their ratio (which is determined for each t) is used for the changepoint detection 206 in the CUSUM algorithm (ie algorithm for changepoint detection with CUSUM statistics), where S:=log(P1(c^i+1|c^i)P0(c^i+1|c^i)) which indicates the log-likelihood ratio. In other words, it detects whether there is a jump in the way the encoded states move between clusters.

[0082] Algorithm 1 gives an example in pseudocode for change detection 206 including clustering 204 and transition dynamics estimation 205 (the usual English keywords such as “for”, “do”, “if”, etc. are used). Although Algorithm 1 has a complexity of O((T0+T1)k) it can be optimized to O(1) using online clustering techniques and a windowed version of kernel estimation.

[0083] If the change point detection 206 detects a change, the change point index is increased and the control strategy 207, which operates, for example, on the state representations output by the encoder 201, switches to a different decision rule (and different assumption about the environment), whereby for each decision rule an environment model (or a distribution of the environment parameters) is learned according to a respective BAMDP.

[0084] Thus, the training in Fig.The method presented in Figure 2 uses one decision rule per transition dynamics to quickly achieve good performance under model uncertainty. This approach allows the continuity of the latent space to be managed. At the same time, change detection 206 (together with clustering 204 and transition dynamics estimation 205) addresses the non-stationarity of the MDP using the state samples encoded by encoder 201. As soon as change detection 206 detects a change, the current control strategy parameters (i.e., those that specify the decision rule just used) are stored along with the associated observations.

[0085] Algorithm 2 gives an example in pseudocode for training the control strategy, here specifically using a Soft Actor Critic (SAC) RL procedure (as above, the usual English keywords such as “for”, “do”, “if”, etc. are used). Algorithm 2 1 Input: Initialized SAC agent A1, Transition estimate P 1 and detection threshold D for change 2 Initialization: List of SAC agents A¯:={A1} and estimated transition models P¯:={P1} of length COUNTER = 1 3 for t = 1, ... do 4 Update change detection with new sample 5 if Algorithm 1 returns a change then 6 Determine new transition kernel estimate P 7 if ∥P - P i ∥ > D then 8 COUNTER = COUNTER + 1 9 Add new agent COUNTER+1 to the list of agents A¯ added. 10 end if 11 Set current agent At on the transition dynamics closest to P 12 end if 13 Train the current agent At 14 end for

[0086] In addition to the change detection, Algorithm 2 creates a list A¯ of agent parameters (i.e., a set of decision rules). Algorithm 2 starts with a single agent, and change detection runs in parallel. As soon as a change is detected, the current decision rule (i.e., the current agent) is saved for later reference, depending on whether the estimated transition kernel P has occurred in the past. If it is close enough, i.e., below a threshold D, to a transition kernel that occurred in the past, then the previously trained agent is retrieved; otherwise, training of a new adaptive Bayesian strategy begins.

[0087] In summary, based on the variBAD algorithm, which derives an assumption about task embeddings through a VAE, change detection is performed on the coded observations to detect changes in the distribution of environmental parameters. Combining these two components, a piecewise Bayesian optimal control strategy is learned, whose jumps correspond to the change points. In Algorithm 2, change detection is independent of the training of the control strategy. Numerical experiments show that this approach can achieve rapid adaptation to intra-episodic MDP changes despite the continuity of the task space.

[0088] In summary, according to various embodiments, a method is provided as described in Fig. 3 shown.

[0089] Fig.3 shows a flowchart 300 illustrating a method for training a control strategy for a technical system according to one embodiment.

[0090] The technical system is controlled in 301. • in 302 a sequence of states of the technical system that occur when controlling the technical system; • in 303 the transition dynamics between successive states of the sequence of states are estimated; • in 304 changes in the transition dynamics during the sequence of states are detected using the estimates of the transition dynamics; and • in 305, in response to the detection of a change in the transition dynamics from a first transition dynamics to a second transition dynamics (i.e. a “jump” in the transition dynamics), the decision rule according to which the technical system is controlled and which is trained on the basis of the sequence of states is changed from a first decision rule associated with the first transition dynamics to a second decision rule associated with the second transition dynamics.

[0091] In other words, according to various embodiments, a sequence of states is decomposed into subsequences based on the detection of changes in the transition dynamics, and a respective (instance of a) decision rule is trained for each subsequence (whereby this may be the same for subsequences if the subsequences have the same transition dynamics). Thus, the subsequences are expressed as multiple classes of control environmental conditions, and a decision rule for the control strategy is trained for each class of control environmental condition.

[0092] The procedure of Fig.3 may be performed by one or more computers having one or more data processing units. The term “data processing unit” may be understood as any type of entity that enables the processing of data or signals. The data or signals may, for example, be handled according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit may include or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA) integrated circuit, or any combination thereof.Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0093] According to various embodiments, the method is therefore particularly computer-implemented.

[0094] The approach of Fig.3 serves, for example, to generate a control signal for a robotic device. The term "robotic device" can be understood as referring to any technical system (with a mechanical part whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. A control rule for the technical system is learned, and the technical system is then controlled accordingly.

[0095] The general term "agent" is used herein, specifically, to refer to all types of technical systems that can be controlled using the approaches described herein. However, the approaches described herein can be applied to any type of agent (e.g., even to an agent that is only simulated and does not physically exist).

[0096] Various embodiments may receive and use sensor signals from various sensors, such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc., for example, to obtain sensor data regarding states of the technical system and / or the environment (robot and object or objects), configurations, and scenarios. The sensor data may be processed. This may include classifying the sensor data or performing semantic segmentation on the sensor data, for example, to detect the presence of objects (in the environment in which the sensor data was obtained). Embodiments may be used to train a machine learning system and control a robotic device, e.g., vehicles or robot manipulators, to achieve various tasks under different scenarios.

Claims

[1] Method for training a control strategy (207) for a technical system (101), comprising: Controlling the technical system (101) and determining a sequence of states of the technical system (101) that occur during the control of the technical system (101); Estimating the transition dynamics between successive states of the sequence of states; Detecting changes in the transition dynamics over the course of the sequence of states based on the estimates of the transition dynamics; and in response to the detection of a change in the transition dynamics from a first transition dynamics to a second transition dynamics, Changing the decision rule according to which the technical system (101) is controlled and which is trained on the basis of the sequence of states from a first decision rule associated with the first transition dynamics to a second decision rule associated with the second transition dynamics. [2] The method of claim 1, comprising, in response to detecting a further change in the transition dynamics, switching back to the first decision rule in response to a determination that the transition dynamics have changed in the further change to the first transition dynamics. [3] Method according to claim 1 or 2, wherein changing the decision rule to the second decision rule is a resetting of the decision rule to a predetermined initial version. [4] Method according to one of claims 1 to 3, comprising training the first decision rule based on states and state transitions of the sequence of states before the change, wherein the training of the first decision rule comprises learning a first transition dynamics model for the first transition dynamics and the first decision rule is trained using the first transition dynamics model and / or comprising training the second decision rule based on states and state transitions of the sequence of states after the change, wherein the training of the second decision rule comprises training the second decision rule, learning a second transition dynamics model for the second transition dynamics and the second decision rule is trained using the second transition dynamics model. [5] Method according to one of claims 1 to 4, wherein the transition dynamics are estimated by encoding states of a subsequence of the sequence of states into a sequence of state representations in a latent space (202) by means of an autoencoder (201), and the transition dynamics for the states of the subsequence are estimated in the form of a transition kernel between elements of the latent space. [6] The method of claim 5, wherein each state representation of the sequence of state representations is assigned to a cluster and transition dynamics are estimated in the form of a transition kernel between elements of the latent space representing the clusters to which the state representations are assigned. [7] The method of claim 6, wherein a first transition kernel is estimated for a first subsequence of the sequence of states and is estimated for a second subsequence of the sequence of states following the first subsequence, and a change in the transition dynamics is detected based on a ratio of the second transition kernel to the first transition kernel. [8] Method according to one of claims 1 to 7, wherein the first decision rule and the second decision rule are trained by means of reinforcement learning. [9] Method for controlling a technical system (101), comprising: Training a control strategy (207) with a plurality of decision rules, each associated with a transition dynamic, according to one of claims 1 to 8; and Controlling the technical system (101) according to the trained control strategy (207), wherein changes in the transition dynamics are detected and, in response to the detection of a change in the transition dynamics, the decision rule according to which control is carried out is changed to a decision rule of the control strategy (207) which is associated with the transition dynamics to which the transition dynamics change. [10] Method according to one of claims 1 to 9, wherein the technical system (101) is the brake (106) of a vehicle (101). [11] Control device (102) which is arranged to carry out a method according to one of claims 1 to 10. [12] A computer program comprising instructions which, when executed by a processor (103), cause the processor (103) to perform a method according to any one of claims 1 to 10. [13] A computer-readable medium storing instructions that, when executed by a processor (103), cause the processor (103) to perform a method according to any one of claims 1 to 10.

Images