Method for training a control strategy for a technical system

Abstract

According to various embodiments, the invention provides a method for training a control strategy for a technical system, comprising controlling the technical system and determining a sequence of states of the technical system that occur when the technical system is being controlled, estimating the transfer dynamics between successive states of the sequence of states, detecting changes in the transfer dynamics over the course of the sequence of states based on the estimates of the transfer dynamics and, in response to detecting a change in the transfer dynamics from first transfer dynamics to second transfer dynamics, changing the decision rule in accordance with which the technical system is controlled and that is trained based on the sequence of states from a first decision rule associated with the first transfer dynamics to a second decision rule associated with the second transfer dynamics.

Description

[0001] R. 406097 Description Title The present disclosure relates to methods for training a control strategy for a technical system. Reinforcement learning is an effective method for learning control strategies for technical systems, such as robots or vehicles. However, this is typically based on the assumption that the transition dynamics, i.e. the rules according to which state transitions of the controlled system (and possibly its environment, such as objects manipulated by a robot arm) occur in response to selected control actions, remain (at least substantially) the same over the course of the control. However, this assumption is not justified for certain control systems, such as an ABS (anti-lock braking system) control of a vehicle, where the transition dynamics can change abruptly and considerably, since, for example, wheels lock much more easily on a dirt road than on asphalt.Therefore, approaches are desirable that enable the training of a control strategy (and effective control) in applications where the transition dynamics can change abruptly and significantly. According to various embodiments, a method for training a control strategy for a technical system is provided, comprising controlling the technical system and determining a sequence of states of the technical system that occur during the control of the technical system, estimating the transition dynamics between successive states of the R. 406097 sequence of states (ieEstimating a transition dynamics for several state transitions (between consecutive states in the sequence of states), detecting changes in the transition dynamics over the course of the sequence of states based on the estimates of the transition dynamics, and, in response to the detection of a change in the transition dynamics from a first transition dynamics to a second transition dynamics, changing the decision rule (or “partial control strategy”, where the control strategy comprises several such partial control strategies) according to which the technical system is controlled and which is trained based on the sequence of states, from a first decision rule associated with the first transition dynamics to a second decision rule associated with the second transition dynamics. The method described above enables the training of a control strategy that is capable of detecting and adapting to changes in the transition dynamics, i.e.to switch to a new decision rule (or to switch to an earlier decision rule if the detected transition dynamics already existed at a previous point in time). This is realized, for example, using three components: an identifier for the dynamics (e.g., using a detection of changes in the dynamics using a cumulative sum ("CUSUM")), an RL (Reinforcement Learning) agent (which learns, for example, using Soft Actor Critic), and a memory for properties of previous environments (i.e., in particular, environmental dynamics). According to various embodiments, the procedure described above can be viewed as the training or subsequent use based on a piecewise Bayesian adaptive decision process and is therefore also referred to below as the PS-BAMDP method (for Piece-Wise Bayesian Adaptive Markov Decision Process). The control strategy (e.g.,a corresponding control device, such as an ABS controller) R. 406097 contains a component that, during its interaction with the environment (whose properties can change abruptly, for example, when changing lanes), observes the properties of the environment and attempts to detect whether there are changes in these properties, in particular the transition dynamics (i.e., in the way states of the controlled technical system and / or the environment) change depending on control actions. Since the input regarding the states from the state space can be continuous (e.g., vehicle speed values ​​can occur in a continuous range), this component clusters the states (for a specific time window) and, based on the clustering, creates a transition matrix (for the time window). If it detects changes in the transition matrix (from time window to time window), it displays (e.g.triggers a signal) that a change in the transition dynamics has occurred. Furthermore, based on the transition matrix (after a change), it tries to understand whether a transition dynamic represented by such (or at least a similar) transition matrix has already occurred in the past (i.e., whether the control strategy has already (at least partially) learned a decision rule for this transition dynamic). If so, it calls the decision rule for this transition dynamic and continues RL learning for it (or uses it for the control if it is already fully trained). If not, a new decision rule is initialized and trained to deal with this new type of transition dynamic. This enables the training of a control strategy that can deal with several different transition dynamics, whereby it can retrieve decision rules for those transition dynamics for which it has already been trained.This means that no large (contextual) decision rule is required (whose context is derived from the detected transition dynamics) to handle the various transition dynamics. Furthermore, this approach provides a certain degree of explainability, since the decision rule used reveals which transition dynamics the control strategy (or a particular control device) currently believes it is interacting with. A R. 406097 decision rule can be understood as a mapping from state to control action (although a certain degree of randomness may be incorporated into this mapping (for the purpose of exploration). The approach described above can be used for any control system where the control device does not know in advance which transition dynamics it will be confronted with.ABS control is a good example of this, as typically an ABS system is tuned to certain, limited scenarios that are defined during its manufacture. The above procedure, however, allows it to adapt to new scenarios as it interacts with these new scenarios. Various embodiments are given below. Embodiment 1 is a method for training a control strategy for a technical system as described above. Embodiment 2 is a method according to embodiment 1, comprising, in response to the detection of a further change in the transient dynamics, switching back to the first decision rule in response to a determination that the transient dynamics have changed during the further change to the first transient dynamics. This ensures that when original conditions are restored, ie a vehicle e.g.first drives on asphalt, then on a dirt road and then back on asphalt, on the asphalt good control results can be achieved immediately again because training for this has already taken place beforehand. For example, it is determined that the transition dynamics have changed during the further change from the first transition dynamics if the transition dynamics after the further change differ from the first transition dynamics by less than a predefined threshold value (in the sense of a predefined metric). R. 406097 For example, a set of decision rules can be defined, where each decision rule is assigned to a respective transition dynamic and where the first decision rule is contained in the set and the second decision rule is added to the set of decision rules if it is not already contained in the set of decision rules.Embodiment 3 is a method according to embodiment 1 or 2, wherein switching the decision rule to the second decision rule is a resetting of the decision rule to a predefined initial version. For example, the second decision rule is a reset version of the first decision rule, based on which the first decision rule was trained. In other words, if it is determined that the transition dynamics have changed and thus the training as performed for the first decision rule is presumably inadequate, it is possible to switch back to an initial version of the decision rule.Embodiment 4 is a method according to one of embodiments 1 to 3, comprising training the first decision rule based on states and state transitions of the sequence of states before the change, wherein the training of the first decision rule comprises learning a first transition dynamics model for the first transition dynamics and the first decision rule is trained using the first transition dynamics model, and / or comprising training the second decision rule based on states and state transitions of the sequence of states after the change, wherein the training of the second decision rule comprises training the second decision rule, learning a second transition dynamics model for the second transition dynamics and the second decision rule is trained using the second transition dynamics model. For example, a critical (for the first orsecond decision rule) use the environmental dynamics model to evaluate the actions chosen by an actor (for the first R. 406097 or second decision rule). The first environmental dynamics model and the second environmental dynamics model can be adapted to the respective state transitions according to Bayesian inference (see explanations below in connection with BAMDPs). Embodiment 5 is a method according to one of the embodiments 1 to 4, wherein the transition dynamics is estimated by coding states of a subsequence of the sequence of states (possibly together with associated actions and / or rewards, i.e. the coding and thus also the estimation of the transition dynamics can depend on rewards and actions that bring about the state transitions) into a model determined by means of a (e.g.variational) autoencoders are encoded into a sequence of state representations in a latent space, and the transition dynamics for the states of the subsequence are estimated in the form of a transition kernel between elements of the latent space. The use of encoding reduces the complexity of change detection. The autoencoder used for encoding can be part of the respective decision rule (i.e., the decision rule according to which control is currently taking place, i.e., the first decision rule for states before the change). For example, a planning module of the respective decision rule can select an action based on the encoding of a respective state. The autoencoder can be trained together with the respective decision rule and also reset if necessary.Embodiment 6 is a method according to embodiment 5, wherein each state representation of the sequence of state representations is assigned to a cluster, and transition dynamics are estimated in the form of a transition kernel between elements of the latent space that represent the clusters to which the state representations are assigned. The use of clusters enables robust change detection. R. 406097 Embodiment 7 is a method according to embodiment 6, wherein a first transition kernel is estimated for a first subsequence of the sequence of states (as described above for the subsequence) and for a second subsequence of the sequence of states (as described above for the subsequence) that follows the first subsequence, and a change in the transition dynamics is detected based on a ratio of the second transition kernel to the first transition kernel (or difference between the two).The first subsequence and second subsequence, corresponding to different time windows, for example, and the transition kernel are each estimated for each of them, and the change is detected by means of change point detection. This enables reliable detection of changes. Embodiment 8 is a method according to one of embodiments 1 to 7, wherein the first decision rule and the second decision rule (and optionally further decision rules of the control strategy) are trained by means of reinforcement learning. The use of reinforcement learning enables effective training of the control strategy (i.e., its various decision rules), particularly in the context of the procedure described above, in which the transition dynamics can be assumed to be constant for each decision rule, since if they change, the decision rule is changed.Embodiment 9 is a method for controlling a technical system, comprising: training a control strategy with a plurality of decision rules, each associated with a transition dynamic, according to one of the embodiments 1 to 8, and controlling the technical system according to the trained control strategy, wherein changes in the transition dynamic are detected and, in response to the detection of a change in the transition dynamic, the decision rule according to which control is carried out is changed to a decision rule of the control strategy R. 406097, which is associated with the transition dynamic to which the transition dynamic changes. Embodiment 10 is a method according to one of the embodiments 1 to 9, wherein the technical system is the brake of a vehicle. For example, the controller is an ABS controller. Embodiment 11 is a control device (e.g.Vehicle dynamics controller) that is configured to perform a method according to any one of embodiments 1 to 10. Embodiment 12 is a computer program having instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 10. Embodiment 13 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 10. In the drawings, like reference numerals generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead being generally placed on illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Figure 1 shows a vehicle.Figure 2 illustrates the training of a control strategy. R. 406097 Figure 3 shows a flowchart illustrating a method for training a control strategy for a technical system according to one embodiment. The following detailed description refers to the accompanying drawings, which, for purposes of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects. Various examples are described in more detail below. Figure 1 shows a vehicle 101.In the example of Figure 1, a vehicle 101, for example a car or truck, is provided with a vehicle control device 102. The vehicle control device 102 has data processing components, e.g., a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software according to which the vehicle control device 102 operates, and data processed by the processor 103. For example, the stored control software (computer program) has instructions which, when executed by the processor, cause the processor 103 to implement a machine learning model 105. The vehicle control device 102 can perform various control tasks ranging from driver assistance functions to fully autonomous driving. The vehicle control device 102 determines the respective control actions using the machine learning model 105 (i.e., the learning model 105 implements a control strategy).For example, according to one embodiment, the vehicle control device 102 is a control device for an anti-lock braking system (ABS), i.e. it generates a desired brake pressure (also referred to as input (wheel) brake pressure) for a brake 106. The vehicle control device 102 (or the control strategy implemented by it) is trained, for example, using reinforcement learning (RL). The training of the vehicle control device 102 can consist of the machine learning model 105 being trained on an external device (training device or training (computer) system) and then being loaded onto the vehicle control device 102, e.g. using simulations (alternatively, the vehicle 101 itself can be implemented as a simulated vehicle in a simulation environment, and the vehicle control device 102 can be trained in this way). In reinforcement learning, an attempt is made to train an agent (ora control strategy that the agent follows), in the above example, to train the vehicle control device 102 (or the machine learning model 105) such that a certain reward is as large as possible (in the above example, for example, depending on how successfully braking was performed). Many reinforcement learning methods typically assume that the environment is stationary. In many realistic cases, however, it can happen that the dynamics of the environment or the reward for the agent changes abruptly during a trajectory (i.e., a sequence of states, possibly with respective rewards and actions) and therefore a rapid adaptation of the control strategy is necessary to ensure safety and performance. This is especially true if the set of R. 406097 possible environments is continuous, since the agent can then no longer accurately identify the true environment.For example, in the case of the ABS control mentioned above, the controller 102 performs braking in different ways depending on the road on which the vehicle 101 is traveling. For example, it could be a concrete road, a sandy road, etc. The dynamics of each road are quite different with respect to the ABS system, and the transition between the different road types is abrupt (e.g., the driver turns from a concrete road onto a sandy road). For example, to be optimally safe and decide when to activate the ABS, the agent (i.e., the controller 102 in the above example) must be precisely tuned to the tire dynamics and the interactions between the road and the tires. However, both depend on the type of road and also weather conditions, two characteristics that can change over time. In the above control scenario, the environment is determined by non-stationary, (from the agent's perspective) unobserved parameters.It can be modeled as a partially observable Markov decision process (POMDP), where the transition function (i.e., the transition dynamics) and the reward function represent the unobserved state. However, in contrast to the general POMDP, where the hidden state changes at each step, in the above control scenario the hidden observations vary at irregular time intervals (e.g., when changing lanes). Instead of constantly maintaining an assumption about hidden states, according to various embodiments, this additional structure (change points at different time intervals) is exploited and an MDP model is derived between each change point. R. 406097 A non-stationary Markov decision process (MDP) is a tuple (S, A, γ, p, r), where S and A are the state and action space, respectively, and γ ∈ [0, 1] is a discount factor.Each element of the state space is a state of the respective controlled technical system (which can include state information about the environment of a controlled device, e.g., when a robot manipulates objects). In the above example of ABS control, for example, this is a state of the vehicle (speed, information about whether the wheels are locked, etc.). Actions from the action space, in the above example, are, for example, different braking forces. The transition parameter p := (p. t (・|s, a)) t∈ℕ is a collection of transition kernels, ie, p t (・|s, a) ∈ Δ S represents a probability over S for all t ∈ ℕ, s ∈ S, a ∈ A. The reward parameter r := (r t (s, is a collection of real-valued functions (s, a) ↦ r t (s, a) ∈ ℝ, which model the immediate performance of the agent. At time t ∈ ℕ, the agent and its environment are in state s tand he carries out the action a t according to a decision rule π t : S → Δ A (ie maps each state to a probability distribution over the actions; here the control strategy contains different decision rules π t for different points in time, e.g., depending on the lane on which the vehicle 101 is currently moving). The collection of decision rules π := ℕ defines the control strategy. The control strategy is trained with the goal of maximizing the expected discounted sum of the This quantity is the value function of the non-stationary MDP. In general, p and r are unknown, and the decisions must be made over an infinite time horizon. This is an extension of the model-based case with a finite time horizon, where dynamic programming can account for non-stationary model parameters. R. 406097 Furthermore, the time dependence of p and r prevents the control strategy from being optimized using standard Bellman operators, since these are based on stationarity. If (p, r) is constant for all t ∈ ℕ, the MDP is stationary and is denoted by (S,A, γ, p, r). Similarly, a control strategy π is stationary if π t=: π for all t ∈ ℕ. It is then simply denoted by π. A consideration of such control strategies is sufficient for stationary MDPs. In complex environments, the transition function and the reward function can be completely represented by abstraction with (relatively) few parameters. Let θ := (θ t ) t∈ ℕ , where, for all t ∈ ℕ, θ t belongs to a parameter space Θ such that (p θ , r θ ) := (p θt , r θt ) t ∈ ℕ eineApproximation of (p, r). According to various embodiments, such an approximation is used for a representation of an MDP model by a neural network (θ then represents, for example, the weights of the neural network, or of several neural networks, namely one for each transition dynamic (and possibly also each reward function), indexed by the t). According to various, in a similar way, a parametric control strategy of the form considered (where ψ is, for example, a neural network that implements the control strategy (or multiple neural networks for different transition dynamics)). Since an accurate determination of the true environmental conditions (in particular the transition dynamics) may be impossible, especially when the parameter space (as in ABS control) is continuous, a Bayesian reinforcement learning (RL) approach is used according to various embodiments. In a Bayesian adaptive MDP (BAMDP), the agent has a probability distribution over possible environmental conditions and uses newly acquired information to reduce the uncertainty regarding the environmental conditions. R. 406097 For this purpose, it is assumed that the transition function and the reward functions follow a priori distribution b(・) over possible (transition and reward) models (p, r), and this assumption (i.e.The distribution across possible models is updated based on new experience. a trajectory that the agent has experienced up to time t. The agent then updates its belief using Bayes' rule, i.e. . Thus, the uncertainty regarding the transition dynamics (and the rewards) can be taken into account by having the agent make decisions based on the current assumption about the transition dynamics (and the rewards). Given a stationary MDP (S, A, γ, p, r), a BAMDP is defined by (S + ,A, γ, p + , r + ), where S + := S × B and B is a set of distributions (for the possible models (p, r)). The extended (transition and reward) model (p + , r + ) is defined as: for all hyperstates (s, b), (s′, b′) ∈ S+ and a ∈ A. In this case, a function π defines + : S + → ΔA a Bayesian adaptive control strategy trained with the goal of maximizing its value function: Theoretically, BAMDPs can be solved using Bellman equations in a hyperstate space. The variBAD algorithm uses variational inference to scale a Bayesian adaptive control strategy to large problems. An amortized inference network (encoder) approximates q ϕ (・|τ :t ) the belief b(・|τ :t ), so that the encoder can be used to simulate R. 406097 new trajectories in the resulting BAMDP and derive the reconstruction loss, i.e., the objective to be minimized by the associated decoder. Furthermore, variBAD trains a strategy π ψ(a|s, qϕ(・ |τ:t)), which is based on the approximate assumption to maximize the return. Although Bayes-optimal strategies solve a single unknown task with low sample complexity, they do not consider the temporal evolution of the dynamics. The BAMDP formalism is based on a stationary but unknown MDP. Therefore, extending the original state space by the belief is not about coping with model changes, but rather about quickly achieving good performance despite model uncertainty. To enable control in environments whose properties can change (so that, in particular, the transition dynamics change), a control strategy based on piecewise stationary Bayes-adaptive Markov decision processes (PS-BAMDPs) is trained according to various embodiments.A PS-BAMDP is a sequence of stationary BAMDPs and can be seen as an extension of BAMDPs to time-dependent MDPs. The control strategy can be trained individually for each BAMDP in the sequence (e.g., as described above using variBAD). The stationary BAMDPs that change the PS-BAMDP differ in their latent parameters, i.e., over the course of the PS-BAMDP, the latent parameters evolve according to a renewal process. In the ABS control application, for example, one BAMDP corresponds to an asphalt road and another to a dirt road, i.e., the latent parameters represent the different road surface properties that the control device cannot directly observe. According to various embodiments, the goal of training (here reinforcement learning) is thus to train a non-stationary Bayesian optimal control strategy that achieves an optimal balance between exploration and exploitation.exploitation) and adapts quickly to abrupt changes (especially transition dynamics). For this purpose, according to various embodiments, an algorithm is used, e.g., referred to as the Bayes-Adaptive Quickest Change Detector (BAQCD), which detects changes in the environment by detecting change points and learns Bayes-optimal strategies accordingly. In addition to coping with the continuity of the parameter space, this achieves high performance and adaptability. In the following, PS-BAMDPs are described in more detail. First, a non-stationary BAMDP (NS-MDP) is defined: Let (S,A, γ, p, r) be a non-stationary MDP. A non-stationary BAMDP (NS-BAMDP) is a tuple (S. + ,A, γ, p + , r + ), where p + := t∈ℕ and where for all t ∈ N (S + ,A, γ, p t + , r t +) is a BAMDP. For an NS-BAMDP, a control strategy is represented by a sequence of adaptive Bayesian control strategies π + := (π t +) t ∈ ℕ given the should maximize. Similar to non-stationary MDPs, there is no systematic procedure for optimizing this objective function. In practice, however, it can happen that environmental parameters remain constant for an (unknown) period of time until they undergo a change (as in ABS control, where the vehicle remains on the same roadway for a while). Therefore, if environmental parameters change rarely enough, one can use the data (observations, e.g., braking behavior for a specific roadway) between the change times and train a respective decision rule (i.e., a decision rule for the respective transition dynamics) within such a time interval (between two change times). R. 406097 For this purpose, a piecewise BAMDP (PS-BAMDP) is defined as follows: Let (λ k ) k ∈ ℕ an increasing sequence of positive integers. A piecewise stationary BAMDP (PS-BAMDP) is an NS-BAMDP (S + ,A, γ, p+ , r + ), for which for all t ∈ ℕ holds: The λ k are the change points, i.e., the points in time at which the transition dynamics change (so significantly that a different decision rule is used). The change points are generally unknown and are therefore derived from state samples. This can be done using change point detection (CDP) or Quickest Change Detection (QCD) as follows: Sei (X t ) t ∈ ℕ a sequence of samples (X t ) t ∈ ℕ , such that for all t ∈ ℕ, X t gemäß f ϕ t (・|X t-1 ) is generated, where ∈ Φ is the underlying (transition dynamics) parameter. Furthermore, let k∈ ℕ be a counting index (as in the PS-BAMDP definition above). In CPD (or QCD) it is assumed that := ϕ k up to an unknown step λ k∈ ℕ remains constant and then changes. From this step remains on = ϕ k+1 to the next change point λ k+1 > λ k etc. In other words: For all t ∈ ℕ, X t+1 The goal of CPD is to find all change points (λ k ) k∈ℕ quickly. In practice, it is a matter of finding a suitable stochastic process S := (S t ) t∈ℕ to design a stop time of the form indicates a change point. The process S is referred to as a statistic and the parameter h as the detection threshold R. 406097. When developing a suitable statistic, the delay in detection must be traded off against the frequency of false alarms. Typically, the “CUSUM” statistic and the “Shiryaev” statistic are used because they are optimal under certain conditions. The CUSUM statistic is defined as CUSUM(S, x) := (S + x)+ and the Shiryaev statistic as SHIRYAEV(S, x) := ex(1 + S). Regarding the detection of changes in PS-BAMDPs, in the Bayesian framework, a change point is a random variable with values ​​in ℕ with a geometric prior distribution. According to various embodiments, (λ k ) k∈ℕ considered as a renewal process, ie for all k∈ℕ the change point increments λ k+1 - independently and identically distributed according to a geometric distribution of the parameter p. Thus, the probability that a change occurs is zero at each change point and increases with time until the next change. This is particularly well-suited to the context of ABS systems, as it means that the control conditions remain stable for a certain period of time, while the probability that they change increases with time. The following describes how changes are detected in PS-BAMDs. Between two changes, the PS-BAMD is a BAMD, i.e., between two changes, there is, as described above, an assumption about the environment (i.e., the current control conditions), namely a distribution b(.) over the possible transition dynamics (or, more generally, environmental models), which is adapted using Bayes' rule (between two change points). A change is a change in the distribution. to a different distribution which in turn is adapted using Bayes' rule. For this purpose, according to various embodiments (similar to variBAD), a latent representation of the assumption about the environment is used, as described below with reference to Figure 2. R. 406097 Figure 2 illustrates the training of a control strategy. As shown in Figure 1, a currently observed trajectory τ is used to determine the currently valid transition dynamics. :t (e.g. the speed curve of the vehicle and / or its wheels for a currently performed braking operation) is first input into the encoder 201 of a variation autoencoder (VAE). The function implemented by the encoder 201 is Its output is a latent representation i.e., for each state of the trajectory (which comprises a sequence of states and associated rewards), a sequence of probability distribution specifications (here, normal distributions) in a latent space 202, where each probability distribution specification has the form of a respective pair of mean μ and standard deviation σ. The latent representation of a trajectory (which in particular contains a sequence of states) is thus a sequence of latent individual representations (μ, σ), with one (“state”) representation per state (i.e., time index) of the trajectory. The VAE is trained such that its decoder 203 generates from the latent representations of a trajectory τ :t a reconstruction τ^ :^ which corresponds as closely as possible to the trajectory τ :tThe decoder 203 takes as input encoded states of a trajectory in the latent space and decodes them to reconstruct parts of the trajectory. For example, the VariBAD algorithm is used to, given τ :t the next state s t+1 and the next reward r t+1 to reconstruct. The idea can be seen in the fact that if the encoded states make it possible to predict the next transition and the reward, all relevant information about the environment has been encoded. R. 406097 In the latent space 202, a clustering 204 is performed, i.e., latent state representations are grouped into clusters and each cluster is represented by a respective state representative, so that a trajectory can be constructed by assigning the state representations to the clusters by a sequence of state representatives c^ :^ = (μ^ :^ , σ^ :^) is represented. Then an estimate 205 of the transition dynamics (e.g. in the form of a transition kernel or a transition matrix) P(c^ ^^^ |c^ ^ ). Then, a change point detection 206 is performed (e.g., using the CUSUM statistic in the Markov version) to determine whether the transition dynamics have changed compared to a previous transition dynamics (i.e., compared to a transition dynamics determined for a previously observed and similarly processed trajectory). Thus, the encoded data (by the encoder 201 of the VAE) are clustered, resulting in a "predicted" trajectory c^ for a current trajectory. :^ in the latent space 202. This allows the estimation of a transition kernel from one cluster to another using maximum likelihood estimation: Let T0 and T1 be the sizes of two consecutive, non-overlapping time windows W0 and W1 with the indices: It should be noted that the windows, especially the window W0, can extend from a current cluster into another cluster (than the one to which the state representations of the current trajectory belong) if the time index goes back far enough. In the transition dynamics P(c^ ^^^ |c^ ^ ) can therefore c^ ^^^ and c^ ^ belong to different clusters (this can also be the case without there being a jump in the transition dynamics). R. 406097 Based on these windows, a transition kernel P0(c^ ^^^ |c^ ^ ) for i from the window W0 and a transition kernel P(c^ ^^^ |c^ ^ ) for i from the second window. Their ratio (which is determined for each t) is used for the changepoint detection 206 in the CUSUM algorithm (ie algorithm for changepoint detection with CUSUM statistics), where indicates the log-likelihood ratio. In other words, it detects whether there is a jump in the way the encoded states move between clusters. Algorithm 1 provides a pseudocode example for change detection 206, including clustering 204 and transition dynamics estimation 205 (using common English keywords such as "for," "do," "if," etc.). 1 Input: Buffer W0 of size T0; Buffer W1 of size T1; CUSUM threshold h 2 Initialization (CUSUM): S0 =0, G 0 =0, t letzter = 0 3 Initialization (Cluster): K clusters for T c (previous) samples 4 for t = T c +1, … do 5 Order μ t to a cluster (represented by c^ ^ ) 6 Update cluster (if necessary including c^ ^ ) using v on μ t 7Estimate P0(.|.) based on μt-T0-T1:t-T1-18Estimate P1(.|.) based on μ t-T1:t-19 10 11 if then 12 Output: (change point at time t, P 1 (.|.) R. 406097 13 t letzter = t14 end if 15 end for Algorithm 1: Change Detection with CUSUM Statistics Although Algorithm 1 has a complexity of , it can be optimized to O(1) using online clustering methods and a windowed version of kernel estimation. If changepoint detection 206 detects a change, the changepoint index is incremented, and control strategy 207, which operates, for example, on the state representations output by encoder 201, switches to a different decision rule (and different assumption about the environment). For each decision rule, an environment model (or a distribution of the environment parameters) is learned according to a respective BAMDP. Thus, the method shown in Figure 2 trains one decision rule per transition dynamics in order to quickly achieve good performance under model uncertainty. This approach allows the continuity of the latent space to be managed.At the same time, change detection 206 (together with clustering 204 and transition dynamics estimation 205) addresses the non-stationarity of the MDP using the state samples encoded by encoder 201. As soon as change detection 206 detects a change, the current control strategy parameters (i.e., those specifying the decision rule just used) are stored along with the associated observations. Algorithm 2 provides an example in pseudocode for training the control strategy, specifically using a Soft Actor Critic (SAC) RL procedure (as above, the usual English keywords such as "for," "do," "if," etc. are used). R. 406097 1 Input: Initialized SAC agent, transition estimation P. 1 undDetection threshold D for change 2 Initialization: List of SAC agents and estimated transition models of length COUNTER = 1 3 for t = 1, ... do 4 Update change detection with new sample 5 if Algorithm 1 outputs a change then 6 Determine new transition kernel estimate P 7 if then 8 COUNTER = COUNTER + 1 9 Add new agent ZÄHLER+1 zur Listeof agents. 10 end if 11Set current agent to the one for the transition dynamics closest to P 12 end if 13Train the current agent 14 end for Algorithm 2 In addition to change detection, Algorithm 2 manages a list of agent parameters (i.e., a set of decision rules). Algorithm 2 starts with a single agent, and change detection runs in parallel. As soon as a change is detected, the current decision rule (i.e., the current agent) is saved for later reference depending on whether the estimated transition kernel P has occurred in the past. If it is close enough, i.e., below a threshold D to a transition kernel that occurred in the past, then the agent already trained for it is retrieved; otherwise, training of a new adaptive Bayesian strategy begins.In summary, based on the variBAD algorithm, which derives an assumption about task embeddings using a VAE R. 406097, change detection is performed on the coded observations to detect changes in the distribution of the environmental parameters. By combining these two components, a piecewise Bayesian optimal control strategy is learned, whose jumps correspond to the change points. In Algorithm 2, change detection is independent of the training of the control strategy. Numerical experiments show that this approach can achieve rapid adaptation to intra-episodic MDP changes despite the continuity of the task space. In summary, according to various embodiments, a method is provided as shown in Figure 3.Figure 3 shows a flowchart 300 illustrating a method for training a control strategy for a technical system according to one embodiment. The technical system is controlled at 301.In this case, ^ in 302 a sequence of states of the technical system that occur during the control of the technical system; ^ in 303 the transition dynamics between successive states of the sequence of states are estimated; ^ in 304 changes in the transition dynamics over the course of the sequence of states are detected using the estimates of the transition dynamics; and ^ in 305, in response to the detection of a change in the transition dynamics from a first transition dynamics to a second transition dynamics (i.e. a "jump" in the transition dynamics), the decision rule according to which the technical system is controlled and which is trained using the sequence of states is changed from a first decision rule associated with the first transition dynamics to a second decision rule associated with the second transition dynamics. R.406097 In other words, according to various embodiments, a sequence of states is decomposed into subsequences based on the detection of changes in the transition dynamics, and a respective (instance of a) decision rule is trained for each subsequence (whereby this may be the same for subsequences if the subsequences have the same transition dynamics). Thus, the subsequences are expressed in several classes of control environmental conditions, and for each class of control environmental condition, a decision rule for the control strategy is trained. The method of Figure 3 can be carried out by one or more computers with one or more data processing units. The term “data processing unit” can be understood as any type of entity that enables the processing of data or signals. The data or signals can, for example, be processed according to at least one (ieone or more than one) special function performed by the data processing unit. A data processing unit may comprise or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an integrated circuit, a programmable gate array (FPGA), or any combination thereof. Any other manner of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be carried out (e.g., implemented) by a data processing unit through one or more special functions performed by the data processing unit.The method is therefore, according to various embodiments, particularly computer-implemented. R. 406097 The approach of Figure 3 serves, for example, to generate a control signal for a robotic device. The term "robotic device" can be understood as referring to any technical system (with a mechanical part whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. A control rule for the technical system is learned, and the technical system is then controlled accordingly. The general term "agent" is also used herein, in particular, for all types of technical systems that can be controlled using the approaches described herein. However, the approaches described herein can be applied to any type of agent (e.g.(also to an agent that is only simulated and does not physically exist). Various embodiments may receive and use sensor signals from various sensors such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc., e.g., to obtain sensor data regarding states of the technical system and / or the environment (robot and object or objects) and configurations and scenarios. The sensor data may be processed. This may include classifying the sensor data or performing semantic segmentation on the sensor data, e.g., to detect the presence of objects (in the environment in which the sensor data was obtained). Embodiments may be used to train a machine learning system and control a robotic device, e.g., vehicles or robot manipulators, to achieve various tasks under different scenarios.

Claims

R. 406097 Claims 1. A method for training a control strategy (207) for a technical system (101), comprising: controlling the technical system (101) and determining a sequence of states of the technical system (101) that occur during the control of the technical system (101); estimating the transition dynamics between successive states of the sequence of states; detecting changes in the transition dynamics over the course of the sequence of states based on the estimates of the transition dynamics; and in response to the detection of a change in the transition dynamics from a first transition dynamics to a second transition dynamics, changing the decision rule according to which the technical system (101) is controlled and which is trained based on the sequence of states from a first decision rule associated with the first transition dynamics to a second decision rule associated with the second transition dynamics. 2.The method of claim 1, comprising, in response to detecting a further change in the transition dynamics, switching back to the first decision rule in response to a determination that the transition dynamics have changed upon the further change to the first transition dynamics.

3. The method of claim 1 or 2, wherein switching the decision rule to the second decision rule is a resetting of the decision rule to a predetermined initial version. R. 406097 4. The method according to any one of claims 1 to 3, comprising training the first decision rule based on states and state transitions of the sequence of states before the change, wherein the training of the first decision rule comprises learning a first transition dynamics model for the first transition dynamics and the first decision rule is trained using the first transition dynamics model and / or comprising training the second decision rule based on states and state transitions of the sequence of states after the change, wherein the training of the second decision rule comprises training the second decision rule, learning a second transition dynamics model for the second transition dynamics and the second decision rule is trained using the second transition dynamics model. 5.Method according to one of claims 1 to 4, wherein the transition dynamics are estimated by encoding states of a subsequence of the sequence of states into a sequence of state representations in a latent space (202) by means of an autoencoder (201), and the transition dynamics for the states of the subsequence are estimated in the form of a transition kernel between elements of the latent space.

6. Method according to claim 5, wherein each state representation of the sequence of state representations is assigned to a cluster, and transition dynamics are estimated in the form of a transition kernel between elements of the latent space that represent the clusters to which the state representations are assigned.

7. Method according to claim 6, wherein a first transition kernel is estimated for a first subsequence of the sequence of states and is estimated for a second subsequence of the sequence of states that is adjacent to the. R. 406097 first subsequence, and a change in the transition dynamics is detected based on a ratio of the second transition kernel to the first transition kernel.

8. The method according to one of claims 1 to 7, wherein the first decision rule and the second decision rule are trained using reinforcement learning. 9.A method for controlling a technical system (101), comprising: training a control strategy (207) with a plurality of decision rules, each associated with a transition dynamic, according to one of claims 1 to 8; and controlling the technical system (101) according to the trained control strategy (207), wherein changes in the transition dynamic are detected and, in response to the detection of a change in the transition dynamic, the decision rule according to which control is carried out is changed to a decision rule of the control strategy (207) that is associated with the transition dynamic to which the transition dynamic changes.

10. The method according to one of claims 1 to 9, wherein the technical system (101) is the brake (106) of a vehicle (101).

11. A control device (102) that is configured to carry out a method according to one of claims 1 to 10. 12.A computer program comprising instructions which, when executed by a processor (103), cause the processor (103) to perform a method according to any one of claims 1 to 10.

13. A computer-readable medium storing instructions which, when executed by a processor (103), cause the processor (103) to perform a method according to any one of claims 1 to 10.

Images