METHOD AND DEVICE FOR DETERMINING A REGULATION STRATEGY FOR A TECHNICAL SYSTEM

Description

Technical field

[0001] The present invention relates to methods for finding a suitable control strategy for a technical system, in particular using a Bayesian optimization method. In particular, the present invention relates to a computer-implemented method for more efficiently optimizing model parameters of a control model for implementing a control strategy for a technical system. Technical background

[0002] In so-called reinforcement learning, a control system repeatedly interacts with its environment and autonomously learns an action strategy from the system's behavior. The learning process occurs through the minimization of a cost value, defined by a cost function or quality function, which assesses the control system's performance.

[0003] Model-free reinforcement learning methods are particularly advantageous because they require no knowledge of the environment or its interaction with the system. However, the system's interaction time with the environment during the learning process is very high. In contrast, conventional model-based methods initially define a model structure that fundamentally describes the behavior of the control system within its environment. This allows the effects of the control system's interaction with the environment to be modeled. Adjusting the model structure by modifying parameters is simple and efficient. However, the conventional method has disadvantages because the chosen model structure may be unsuitable, and therefore, parameter adjustments through optimization of the control model may not lead to an optimal result.

[0004] The creation and optimization of a control model in conjunction with reinforcement learning can be performed in combination with Bayesian optimization, where the cost function is modeled as a Gaussian process model. This enables the creation of an efficient black-box optimizer for control models that would otherwise be complex to create and parameterize. However, due to the large number of model parameters in such control models, optimization is very complex, requiring a large number of measurement operations, and long training times are the norm due to the large amount of measurement data. US 2013 / 158678 A1 discloses a method for controlling technical equipment. XP080467931 describes Bayesian optimization methods. CN 108 508 856 A discloses an intelligent method for controlling an industrial plant. WO 2018 / 219943 A1 discloses a method for controlling a robot actuator. Disclosure of the invention

[0005] According to the invention, a method for creating a control system for a technical system according to claim 1, as well as the device and a system according to the dependent claims, are provided.

[0006] Further details are specified in the dependent claims.

[0007] According to a first aspect, a procedure for creating a control system for a technical system using a Bayesian optimization method is provided, wherein the control system is created based on model parameters and is executable, and the following steps are carried out to optimize the control system: Providing a quality function whose form corresponds to a trainable regression function and which evaluates the quality of a control of the technical system based on model parameters; performing a Bayesian optimization procedure based on the quality function to iteratively optimize a set of model parameters; wherein, prior to performing the Bayesian optimization procedure, initial control models are created based on several dynamic system models and the model parameter domain, which specifies the permissible ranges of values ​​for the model parameters, is determined by determining a distribution of the model parameters of the initial control models and, depending on an associated maximum a posteriori estimate and their respective standard deviations of the distribution of the model parameters, the ranges of values ​​of the model parameters are initially defined.

[0008] To model the control of a control system, a control model is created. The control model defines a control strategy and represents a functional relationship with which a manipulated variable is calculated based on one or more input variables of the control system. At least one of the input variables corresponds to a state variable of the technical system to be controlled.

[0009] The form of the control model can be defined in a variety of ways. For example, the control model can include or be based on a formula relationship derived from physical laws, such as equations of motion, vibration equations, damping behavior, etc. Alternatively, the control model can be based on a freely chosen formula relationship or one determined in some other way. Often, the formula relationship must account for the nonlinear behavior of the technical system.

[0010] Bayesian optimization methods for determining a control model iteratively apply different control strategies to a technical system and optimize the control efficiently. The quality function can be modeled using a trainable regression function, in particular a Gaussian process regression, to model the performance of the system model as a function of the model parameters of the control model, where the Gaussian process regression is based on noisy (since measured) state variables.

[0011] The fundamental problem is to find a control strategy that maps a system state to an input vector. This involves evaluating the control system's behavior over a predetermined time horizon, based on state vectors (comprising multiple state variables) and input vectors (comprising multiple input variables). This evaluation determines the quality of the system's behavior. The quality of the system's behavior can be assessed by evaluating it against a desired behavior. For example, criteria such as oscillation tendency, settling time, disturbance sensitivity, robustness, and / or similar factors can be evaluated. The control system's behavior is also referred to as an unknown function.The quality function (cost function) corresponds to a mathematical model that evaluates the quality of the control behavior of the technical system depending on the model parameters of the underlying control strategy.

[0012] In general, Bayesian optimization is used when an unknown function f, a so-called "black box" function that corresponds to a behavior of the technical system, is to be minimized. This unknown function f It can only be evaluated or measured for a single value x and observed (possibly subject to noise). The observed value y is then calculated as... y = f(x)+ e, where e denotes the noise component. Furthermore, it is assumed that each evaluation / measurement of the unknown function f is expensive, i.e., incurs costs, in the sense that evaluating the unknown function is very time-consuming, as is the case, for example, when conducting an experiment on a test bench. Due to the expensive evaluation of the unknown function, it is desirable that only a few evaluations need to be performed during optimization.

[0013] Under certain assumptions, such as the continuity of the system behavior or the unknown function, the unknown function can be approximated by the quality function using Gaussian process regression. This involves evaluating the unknown function at several evaluation points ( x 1 , ..., xn ) and observation of the corresponding function values ​​( y 1 , ... ynThe quality function is trained using the Gaussian process. A characteristic of the Gaussian process is that in regions around the evaluation points, the model prediction is very good and the unknown function, i.e., the system behavior, is well approximated. This is reflected in a low uncertainty of the quality function. Far from the evaluation points, the model predictions of the quality function about the unknown function become poor, and the uncertainty increases with increasing distance from the evaluation points.

[0014] One possible strategy for optimizing the model parameters of the control strategy, and thus the system behavior, is to evaluate the unknown function at many different locations (e.g., on a regular grid) and assume the lowest observed function value is the result of the optimization. This approach is inefficient, and finding the optimum requires many evaluations and a correspondingly high workload.

[0015] Instead of this approach, the quality function is used to select new evaluation points. A new evaluation point for assessing / measuring the system behavior is chosen in such a way that, firstly, the predictive power of the quality function is improved, thus reducing the uncertainty of the estimated expected value of the quality function. For this purpose, evaluation points are generally chosen in areas where the unknown function has not yet been assessed or the technical system has not yet been assessed (exploration). Secondly, a new evaluation point for assessing the unknown function or for assessing the technical system is selected in such a way that the goal of finding an optimum is achieved as quickly as possible, i.e., with a small number of measurements at the evaluation points.For this purpose, evaluation points promising low function values ​​are favored for selection based on the Gaussian process (exploitation). These two opposing criteria are balanced by selecting the evaluation point using a predefined acquisition function.

[0016] The acquisition function uses parameters of the quality function, which is described by a Gaussian process model, such as the expected value. µ ( x ) and the standard deviation assigned to this expected value σ ( x ) . One example is the so-called Lower Confidence Bound (LCB) acquisition function, which is described as follows: LCB ( x ) = µ ( x ) - kσ ( x ) .In practice, the factor k is often set to a constant value, such as k = 2. This new criterion can be efficiently minimized using common gradient-based methods, and the minimum of LCB(x) then forms the new evaluation point for the unknown function f, or for measuring the technical system. It is important to note that for optimizing the quality function, a model parameter domain must first be defined, specifying the ranges of values ​​for the respective model parameters within which the next evaluation point is sought. This model parameter domain is typically chosen based on experience and / or expert knowledge.

[0017] After evaluating the behavior of the technical system (e.g., by measuring at the evaluation point), the trainable regression function is updated or retrained with the new data, and a next evaluation point is selected according to the procedure described above. Typically, this process is repeated until a termination criterion is met, such as the time required to optimize the unknown function f (i.e., the behavior of the technical system).

[0018] After the optimization process is completed, the function value x is recommended as the location of the minimum of the quality function, i.e., the value that minimizes the expected value of the Gaussian process.

[0019] The LCB criterion is not the only acquisition function proposed in the literature. Other criteria include, for example, Expected Improvement (EI), Upper Confidence Bound (UCB), Probability of Improvement (PI), or so-called entropy search methods, which are based on information-theoretic considerations.

[0020] According to the procedure described above, Bayesian optimization is used to determine the model parameters of the control strategy by minimizing the initially unknown quality function. The goal is to create the control model so that the behavior of the technical system corresponds as closely as possible to the desired specifications. The quality function can be updated as needed, depending on the performance of the control system and the respective model parameters. Updating the quality function requires an evaluation of the respective model parameters, which necessitates operating the control system with these parameters in the real-world environment. During operation, a quality measure reflecting the performance of the controlled technical system is recorded and assigned to the current model parameters.Because the control system for the surveying process needs to be operated in real time, improving cost evaluation through the continuously updated quality function becomes very complex. Therefore, it is desirable to minimize the number of surveying operations on the actual technical system as much as possible.

[0021] After each of one or more measurement processes, the quality function can be updated with the most recently measured model parameters and the resulting quality measures, and a new set of test model parameters can be selected by minimizing or maximizing a predefined acquisition function that evaluates the course or parameters of the quality function. The acquisition function is predefined to ensure a compromise between exploration and exploitation. In the context of the method according to the invention, exploration means that regions of the possible model parameter space are preferred for the selection of a new test model parameter where a quality measure has not yet been determined or the system behavior evaluated by a previous measurement process.Exploitation in the context of the method according to the invention means that a test model parameter is selected for the next evaluation by a measurement process, which is estimated to be a set of optimal model parameters.

[0022] In the following example, it is assumed that the control strategy is a linear state control strategy of π θ ( x ) = -K ( θ ) x This corresponds to K, where K is a control matrix characterizing the control system. Linear control strategies have the advantage of low dimensionality compared to other control models. Furthermore, the linear control strategy allows for simple implementation in controllers, thus increasing the efficiency of Bayesian optimization.

[0023] Control strategy optimization is performed by adjusting the elements of the control matrix. The elements of the control matrix can be directly assumed to be model parameters or assumed to be dependent on model parameters.

[0024] One idea behind the above method is to determine a control model using a Bayesian optimization procedure within a restricted model parameter domain. While it is not readily possible with a conventional Bayesian optimization procedure to determine the value ranges for individual model parameters within which the search for a control strategy should be performed in such a way that the search is carried out more efficiently, i.e., with fewer measurements, the present method allows meaningful value ranges for the model parameters of the control model to be determined by initially specifying dynamic system models and converting them into initial control models.

[0025] In this way, the model parameter domain for Bayesian optimization can be automatically determined before the optimization procedure, i.e., without manually specifying value ranges for the model parameters, so that the number of measurements for the optimization can be significantly reduced.

[0026] In particular, the control models can be defined by a control matrix K = dlqr(A,B,Q,R). The value ranges for the model parameters can be determined based on a maximum posterior estimate (MAP estimate) of the elements of the control matrix K. Furthermore, the value ranges can be defined based on a tolerance range.

[0027] Furthermore, the model parameters of the control model can be continuously adjusted or adjusted at regular or predetermined times during the control of the technical system based on the above procedure.

[0028] According to another aspect, a device for creating a control system for a technical system using a Bayesian optimization method is provided, wherein the control system is created and executable based on model parameters of a control model, and wherein the device is designed to perform the following steps to optimize the control system: Providing a quality function that corresponds to a trainable regression function and evaluates the quality of a control system of the technical system based on model parameters; performing a Bayesian optimization procedure based on the quality function to iteratively optimize a set of model parameters with model parameters within an initially specified model parameter domain; wherein, prior to performing the Bayesian optimization procedure, initial control models are created based on several dynamic system models and the model parameter domain, which specifies the permissible value ranges for the model parameters, is determined by determining a distribution of the model parameters of the initial control models and, depending on an associated maximum a posteriori estimate and their respective standard deviation of the distribution of the model parameters, the value ranges of the model parameters are initially defined.

[0029] According to another aspect, a control system comprises a technical system and a control unit for regulating the technical system, wherein a control model for providing an input vector dependent on state variables of the technical system is implemented in the control unit, wherein a model creation block is provided to iteratively optimize a model parameter set with model parameters for the control model based on a Bayesian optimization procedure executed in an optimization block within an initially specified model parameter domain, wherein the Bayesian optimization procedure is based on a specified quality function, which is determined in a quality function block.The process is performed, whereby, prior to the execution of the Bayesian optimization procedure, initial control models are created based on several dynamic system models, and the model parameter domain, which specifies the permissible value ranges for the model parameters, is determined by determining a distribution of the model parameters of the initial control models and, depending on an associated maximum a posteriori estimate and their respective standard deviation of the distribution of the model parameters, the value ranges of the model parameters are initially defined. Brief description of the drawings

[0030] The embodiments are explained in more detail below with reference to the accompanying drawings. These show: Figure 1 is a schematic representation of a control system with a control unit and a technical system to be controlled; Figure 2 is a flowchart illustrating the procedure for performing an optimization of a control strategy using a Bayesian optimization method; Figure 3 is a diagram with a schematic representation of the functioning of the dynamic adjustment of the model parameter domain. Description of embodiments

[0031] Figure 1Figure 1 shows a schematic representation of a self-adapting control system 1 designed to control a technical system 2. A technical system 2 could, for example, be an internal combustion engine of a motor vehicle or a subsystem thereof. A control unit 3 controls the technical system 2 with a sequence of input variables u as manipulated variables, which lead to specific operating points (system states) of the technical system 2. The input variables u typically comprise a number of multiple input variables represented in an input variable vector. u ∈ ℝ d are summarized. Furthermore, the following applies to each of the input variables (elements of the input variable vector). u ) a permissible range of values. Furthermore, the control of the technical system 2 results in one or more state variables, which are measured against an input vector. u are measured and are in the form of a state variable vector x are shown.

[0032] Using one or more sensors 21, which are part of the technical system 2, the profiles of one or more measured variables x1 ... xD can be recorded, each representing corresponding state variables x1(t) ... xD(t), which in turn indicate the system states x of the technical system 2. D corresponds to the number of state variables. Thus, the system state of the technical system 2 is recorded using the one or more sensors 21 and transmitted as state variables of a state variable vector x to the control unit 3.

[0033] Input variables u correspond to manipulated variables of the control unit 3, which are based on the state variables x and the control strategy. π θ ( x The operation of the technical system 2 depends on the input variables u(t) and is carried out using one or more actuators 22 of the technical system 2. For example, the movement of a robot or vehicle can be controlled, or a drive unit or a driver assistance system of a vehicle can be controlled. An input variable u can, for example, correspond to an electrical voltage applied to an electromechanical encoder as actuator 22. The actuator 22 is controlled according to the one or more input variables u and performs a corresponding action. The actuator 22 can include a control logic (not necessarily integrated structurally), which is derived from the input variables u A control parameter is determined with which the relevant actuator 22 is controlled.

[0034] In one embodiment, the control unit 3 is used as a technical system for controlling an internal combustion engine. For this purpose, a throttle valve position, fuel supply, and / or the like can be specified as input variables to the throttle valve actuator or the control unit for the injection valves, and corresponding state variables, such as speed, load, and engine temperature, can be received.

[0035] In one embodiment, the control unit 3 is used as a technical system 2 for controlling an at least partially autonomous robot, in particular an at least partially autonomous motor vehicle. The sensor 21 can be, for example, one or more video sensors, preferably arranged in the motor vehicle, and / or one or more radar sensors, and / or one or more ultrasonic sensors, and / or one or more LiDAR sensors, and / or one or more position sensors (e.g., GPS). Alternatively or additionally, the sensor 21 can also include an information system that determines information about a state of the technical system (motor vehicle), such as a weather information system that determines a current or future weather condition in the vicinity of the motor vehicle.

[0036] In another embodiment, the control unit 3 is used as a technical system to control a function in a motor vehicle. For this purpose, input variables can include accelerator pedal position, steering input in the form of a wrist torque or steering angle, environmental information such as the position of surrounding objects, brake application, and / or the like. Corresponding state variables that indicate the vehicle's driving behavior, such as vehicle speed, cornering angle, distance to surrounding objects, and the like, can be received.

[0037] The control unit 3 can use the multiple measured variables x1 ... xD to detect, for example, states or state profiles of the at least semi-autonomous robot, such as engine speed, vehicle speed, fuel consumption, engine temperature, longitudinal and / or lateral speed, steering angle, yaw rate, and the like. The actuator 32, preferably arranged in the motor vehicle, can be, for example, a brake, a drive, or a steering system of the motor vehicle.

[0038] Alternatively, the at least semi-autonomous robot could also be another type of mobile robot (not shown), for example, one that moves by flying, swimming, diving, or walking. The mobile robot could also be, for example, at least a semi-autonomous lawnmower or at least a semi-autonomous cleaning robot.

[0039] In further alternative scenarios, the at least semi-autonomous robot could also be a household appliance (not shown), in particular a washing machine, a stove, an oven, a microwave, or a dishwasher. Sensor 21, for example an optical sensor, can detect the state of an object being handled by the household appliance; for instance, in the case of a washing machine, the state of the laundry inside. Control unit 3 can then determine the type or state of this object and characterize it using the measured variables x1...xD. The input variables can then be determined in such a way that the household appliance is controlled depending on the determined type or state of the object. For example, in the case of a washing machine, it could be controlled depending on the material of the laundry inside.The input variables u(t) can then be chosen depending on which material of the laundry was determined.

[0040] In another embodiment, the control unit 3 can be used to control a production machine (technical system 3) of a production system by controlling an actuator 22 that controls this production machine via input variables. The production machine 11 can, for example, be a machine for punching, sawing, drilling, milling, turning and / or cutting.

[0041] Sensor 21 could, for example, be an optical sensor that detects properties of manufactured products. It is possible that the actuator 22, which controls the manufacturing machine, is activated based on the determined properties of the manufactured product, so that the manufacturing machine can then perform a subsequent processing step on that product. It is also possible that sensor 21 determines the properties of the manufactured product being processed by the manufacturing machine and adjusts the machine's control accordingly for a subsequent product.

[0042] The control unit 3 follows a control strategy. A dynamic process is used to adapt the control strategy so that the system behavior is optimized with respect to a quality function. For this purpose, an optimization procedure is executed that optimizes the model parameters of the underlying control model to improve the performance of the controlled technical system 2. To this end, a control model (dynamic model) is created in a model creation block 4, which forms the basis for the control strategy of control unit 3. Model creation block 4 determines the model parameters for the control model based on a Bayesian optimization procedure executed in an optimization block 5. This is based on a predefined quality function, which is determined or specified in a quality function block 6.

[0043] In further preferred embodiments, the control unit 3, the model creation block 4, the optimization block 5, and the quality function block 6 are implemented in a computing unit. The computing unit comprises the control device 2, one or more processors, and at least one machine-readable storage medium on which instructions are stored which, when executed on the processors, cause the computing unit to execute the method according to the invention.

[0044] Technical system 2 is a dynamic system that is controlled in an optimized manner by a control unit 3 with a suitable control strategy, which is to be developed using a Bayesian optimization method. The Bayesian optimization method is used to determine the control model by iteratively applying various test model parameter sets to the control of technical system 2 during the optimization process and adjusting the model parameters based on the resulting state variables. A quality function is then modeled using Gaussian process regression or another trainable regression method, which defines the performance of the control of technical system 2 as a function of the model parameters.

[0045] The performance of the control system is derived from a predefined quality criterion (quality measure) that assigns a control quality rating to the resulting state variables (especially their behavior) of the technical system 2, based on a comparison with the desired behavior of the technical system 2 controlled by the control model based on the model parameters. The state variables are subject to tolerances, so the quality function is preferably represented by a Gaussian process regression.

[0046] The fundamental problem is finding a control strategy that addresses a system state. x on an input vector u = π θ ( x ) with π θ : ℝ n x → ℝ n u maps, with model parameters of the control strategy in the model parameter domain. Θ represent. One of the model parameters θ The dependent quality function J is calculated over a predetermined time horizon t=0...T based on the state vectors. x and input vectors u specified, where the model parameters θ The following should be optimized through the optimization process: min θ J θ = min θ ∑ t = 0 T E c x t , π θ x t s . t . x t + 1 = f x t , π θ x t + v where corresponds to an expected value c ( xt, ut ) the quality measure of the state vector xt given state with applied input vector ut and f : ℝ n x × ℝ n u → ℝ n x represent the state transition model that describes the dynamics of the technical system 2 and is further defined by the noise quantity ν ~ N (0, Σ ν The quality measure can be determined by comparing the system behavior with a desired system behavior, whereby the quality criterion can specify robustness, susceptibility to vibration, settling time and control quality.

[0047] The Bayesian optimization method is used to optimize the model parameters. θ* A control strategy can be found by minimizing the quality function (which, for example, represents the costs). Alternatively, the optimized model parameters can be used. θ* Depending on the informative value of the quality measures, the control strategy can also be determined by maximizing the quality function. The goal is for the control strategy to enable the most optimal possible control of the technical system 2 with the control unit 3, where "optimal" means minimizing / maximizing (optimizing) a quality measure that is determined by the quality function depending on the model parameters and indicates a predetermined performance level of the overall system consisting of control unit 3 and technical system 2.

[0048] The quality function can therefore specify a quality measure that evaluates the deviation of the behavior of the real technical system 2 during the time window t=0..T relative to a predefined desired performance (ideal system behavior). Evaluating the quality function thus requires operating the technical system 2 in its real environment during a measurement process. Due to the necessity of operating the control system, which comprises the technical system 2 and the control unit 3, in real-world conditions, evaluating the quality measure of the behavior of the technical system 2 becomes very complex. Therefore, the number of measurement processes on the real technical system 2 for evaluating a specific control strategy should be minimized as much as possible.

[0049] It is assumed in the following that the control strategy is a linear state control strategy with π θ ( x ) = -K ( θ ) x corresponds to, where K corresponds to a control matrix whose elements correspond to or depend on the model parameters of the control strategy.

[0050] Linear control strategies have the advantage of exhibiting low dimensionality compared to other control models. Furthermore, linear control strategies can be easily implemented in controllers, thus increasing the efficiency of Bayesian optimization.

[0051] In conjunction with Bayesian optimization, a linear quadratic regulator, also known as an LQR controller, can be used, as is common in the development of control strategies. With an LQR controller, the system behavior and interaction with the environment during surveying processes are determined through controlled operation with a set of varying input vectors and the acquisition of resulting state vectors. The system dynamics are then analyzed according to... f x t u t ≈ A x t + B u t linearized and the costs squared accordingly c x t , u t ≈ x t T Q x t + u t T R u t

[0052] These approximations allow the creation of an LQR feedback matrix in model creation block 4, which represents the dynamic model and is generally referred to as the control matrix. K = dlqr ( A, B, Q, RThe control strategy optimization is performed by directly fitting the elements of the control matrix as model parameters to be optimized (K-learning), where a subset or each entry (element) of the control matrix corresponds to a model parameter for optimization. Alternatively, only those components of the control matrix K that correspond to the system matrices A and B can be considered as model parameters to be optimized, with each entry of matrices A and B corresponding to a model parameter (AB-learning).

[0053] Alternatively, only the matrices Q and R (weighting matrices) can be assumed to be the model parameters to be optimized in the control matrix K (QR learning). In this case, it is sufficient to adjust only the diagonal entries of the weighting matrices, which have the following form: K QR θ = dlqr A , B , Q θ , R θ mit Q θ = diag 10 θ 1 , … , 10 θ n x und R θ = diag 10 θ n x + 1 , … , 10 θ n x + n u

[0054] These methods are called K-Learning, AB-Learning and QR-Learning, according to the relationship of the model parameters to be optimized to the matrices above.

[0055] In Bayesian optimization, the model parameter domain must be defined before the optimization begins in order to obtain a good estimate of the optimized model parameters. Without prior knowledge, however, it is difficult to select the value ranges for the individual model parameters—that is, the model parameter domain—for the optimization. This is essential for an efficient optimization procedure without an excessive number of measurement operations.

[0056] The above procedure therefore involves selecting a suitable search range for the model parameters, thus reducing the number of measurement operations. This is particularly necessary in cases of high dimensionality, where manually setting the value ranges of each model parameter is not readily feasible. Therefore, the value ranges of the model parameters are first determined by learning a distribution over dynamic models, and this distribution is then used to select the value ranges for each model parameter. The distribution is obtained through Bayesian linear regression using recorded data values ​​of the state variable vectors and input variable vectors to create an approximate linear model of the system dynamics. This results in a Gaussian distribution over the models. p vec A , B Data = N vec A , B μ AB Σ AB where µ AB< the maximum posterior (MAP) estimate and where the notation vec(.,.) indicates that matrices A and B have been transformed into a vector.

[0057] By determining the control matrices K = dlqr ( A, B, Q, R For a number of predefined dynamic system models, distributions of the model parameters of the elements of the control matrix K can be determined. These are used to select meaningful value ranges for the model parameters. To obtain a distribution over possible control matrices, NS pairs of ( AWAY ) calculated by the above dlqr function to obtain an example distribution (A posterior distribution) p ( νec ( K )| (AWAY ) 1:ns ) to obtain. The distribution of values ​​is then given by a product of independent normal distributions. p vec K A B 1 : n s ≈ ∏ i = 1 n θ N K i μ i K σ i K

[0058] This approximation allows us to determine value ranges Θ. K< of model parameters around the mean of the distribution and with the width given by the standard deviation of the distribution: Θ K = μ 1 K − βσ 1 K , μ 1 K + βσ 1 K × … × μ n θ K − βσ n θ K , μ n θ K + βσ n θ K

[0059] The scaling parameter β determines the effective size of the value ranges of the model parameters.

[0060] As the scaling parameter β increases, the Bayesian optimization method becomes more flexible in finding a better solution, but this requires more measurement passes to determine the model parameters. Therefore, it is necessary to choose a trade-off between limiting the model parameter ranges to a reasonable size and allowing sufficiently large ranges to avoid errors due to a misfit of the system model. For example, the model parameter ranges can be dynamically adjusted during the optimization process.

[0061] To utilize potential correlations between the model parameters, the distribution of values ​​can be approximated using a multivariable Gaussian process. p(νec ( K )|( AWAY ) 1: n S ) ≈ N(νec ( K )| µ K< , Σ K< Here, it is proposed to place the model parameters in the eigenspace of the covariance matrix. Σ K< with θ̃ = T ( θ - µ K< ) to transform, where the transformation matrix T consists of the eigenvectors Σ K< exists. In the eigenspace, the value ranges of the model parameters then become Θ ˜ PCA K = − βσ 1 K , β σ ˜ 1 K × … × − β σ ˜ n θ K , β σ ˜ n θ K , where σ̃ i ; the i-th eigenvalue of Σ K indicates. Essentially, a basic component analysis (PCA) is performed.

[0062] After selecting the value ranges for the model parameters, it is possible that the value ranges for certain dimensions are chosen too conservatively. This can occur, for example, if the scaling parameter β is too small, or due to a model deviation. Therefore

[0063] During the optimization process, it may happen that the optimal model parameter vector lies outside the selected model parameter domain. Therefore, the model parameter domain can be dynamically adjusted during the optimization.

[0064] During the Bayesian optimization process, an estimated optimum of the model parameter values ​​is obtained, i.e., the minimum of the approximated quality function within the current model parameter domain. If the Bayesian optimization reveals that the estimated optimum lies at a boundary of the model parameter domain, it is likely that better model parameters lie outside the current domain. Therefore, it is proposed to expand the range of values ​​for the model parameter whose value lies at the boundary of the model parameter domain. This dynamic adjustment of the model parameter domain can be implemented in various ways.

[0065] Adjusting the value ranges for model parameters allows for dynamic adjustments during optimization, starting from a limited range, only for those dimensions of the model parameter vector where the optimization encounters a boundary region of the model parameter domain. This makes the overall optimization more efficient, significantly improving convergence. Furthermore, potential model errors can be better compensated for, resulting in a more powerful optimized system model. This improved efficiency enables the scaling of Bayesian optimization to high-dimensional control strategies.

[0066] To illustrate the creation and adaptation of a control system for a technical system, the procedure is described below using a flowchart. Figure 2 described.

[0067] In step S1, a series of system models are specified, each based on physical model equations, and the control matrix K=dlqr(A,B,Q,R) is determined according to the procedure described above. The system models preferably have different parameterizations.

[0068] The distribution of the initial control models is obtained by creating a Gaussian distribution over each of the model parameters of the control matrix K. p vec A B Data = N vec A B μ AB Σ AB where µ AB< the maximum posterior (MAP) estimate (expected value of the corresponding Gaussian distribution) and the notation vec (.,.) indicates that matrices A and B have been transformed into a vector.

[0069] In step S2, the distributions are used to calculate the maximum posterior estimate (MAP estimate) of the elements. θ MAP< of the control matrix K is determined. The maximum posterior estimate (MAP estimate) of the elements θ MAP< of the control matrix K corresponds p w t = N w m n , S n , θ MAP = m n = S n S 0 - 1 m 0 + γ X T t , S n − 1 = S 0 − 1 + γ X T X where n is the number of data points, γ a given constant that describes the noise in the data, m 0 an a priori mean of the model parameters to be estimated, S 0 an a priori covariance of the model parameters to be estimated, X the locations of the data points (summarized in a matrix), t the function values ​​of the data points (summarized in a vector), mn the a posteriori expected values ​​(means) of the model parameters to be estimated and S n correspond to the a-posteriori covariance of the model parameters to be estimated.

[0070] The distribution of values ​​is then given by a product of independent normal distributions. p vec K A B 1 : n s ≈ ∏ i = 1 n θ N K i μ i K σ i K where ns the number of samples determined by p(vec(K) | A, B), and n θ correspond to the number of parameters to be estimated.

[0071] This approximation allows us to determine value ranges Θ. K< of model parameters around the mean of the distribution and with the width given by the standard deviation of the distribution: Θ K = μ 1 K − βσ 1 K , μ 1 K + βσ 1 K × … × μ n θ K − βσ n θ K , μ n θ K + βσ n θ K

[0072] The scaling parameter β determines the effective size of the value ranges of the model parameters.

[0073] As the scaling parameter β increases, the Bayesian optimization method becomes more flexible in finding a better solution, but this requires more measurement passes to determine the model parameters. Therefore, it is necessary to choose a trade-off between limiting the model parameter ranges to a reasonable size and allowing sufficiently large ranges to avoid errors due to a misfit of the system model. For example, the model parameter ranges can be dynamically adjusted during the optimization process.

[0074] To utilize potential correlations between the model parameters, the distribution of values ​​can alternatively be approximated using a multivariable Gaussian process. p(νec ( K )|( AWAY ) 1:ns ) ≈ N ( νec ( K )| µ K< , Σ K< Here, it is proposed to place the model parameters in the eigenspace of the covariance matrix Σ. K< with θ̃ = T ( θ - µ K< ) to transform, where the transformation matrix T consists of the eigenvectors Σ K< exists. In the eigenspace, the value ranges of the model parameters then become Θ ˜ PCA K = − βσ 1 K , β σ ˜ 1 K × … × − β σ ˜ n θ K , β σ ˜ n θ K , where σ̃ i the i-th eigenvalue of Σ K indicates. Essentially, a basic component analysis (PCA) is performed.

[0075] In step S3, the value ranges for the control parameters to be determined for the control model are defined. Bayesian optimization must cover the value ranges of the control parameters as adequately as possible, also considering the length scales of the cost function. A priori, it is often difficult to determine in which value ranges the control parameters, i.e., the entries of the control matrix K, must be optimized to achieve good control performance.

[0076] For this purpose, MAP estimates are used. θ MAP< From the system identification of step S2, meaningful value ranges are selected for each of the control parameters, within which the control parameters are subsequently optimized.

[0077] In step S4, the optimization process is started.

[0078] A new set of test model parameters will be used for this purpose. θ ' selected by a predefined acquisition function α ( θ; D n ) with D n+ 1 is maximized or minimized. The acquisition function α ( θ; D nThe process is designed to utilize a combination of exploration and exploitation to determine a new set of test model parameters at a minimum or maximum (depending on the chosen acquisition function) of the acquisition function. The acquisition function uses parameters of the quality function created as a Gaussian process model, such as the Gaussian process mean (expected value) and the Gaussian process standard deviation. In the context of the method according to the invention, exploration means that regions of the possible model parameter space are preferred for the selection of a new test model parameter where the behavior of the technical system has not yet been evaluated by previous measurement processes.Exploitation in the context of the method according to the invention means that a set of model parameters is selected for the next evaluation of the behavior of the technical system by a measurement process, which is considered to be a set of optimal model parameters. θ * = arg min θ J θ ; θ ′ ← θ *

[0079] The acquisition function uses parameters of the quality function, which is described by a Gaussian process model, such as the expected value. µ ( x ) and the standard deviation assigned to this expected value σ ( x ) . One example is the so-called Lower Confidence Bound (LCB) acquisition function, which is described as follows: LCB ( x ) = µ ( x ) - kσ ( x ) .This new criterion can be efficiently minimized using common gradient-based methods, and the minimum of LCB(x) then forms the new evaluation point for the unknown function f or the measurement of the technical system 2.

[0080] The LCB criterion is not the only acquisition function proposed in the literature. Other criteria include, for example, Expected Improvement (EI), Upper Confidence Bound (UCB), Probability of Improvement (PI), or so-called entropy search methods, which are based on information-theoretic considerations.

[0081] In step S5, a measurement process is carried out and, based on the test model parameter set, the control of the technical system 2 by the control unit 3 is evaluated during the time window t=0...T in order to record test data in the form of input variables u and resulting state variables x.

[0082] Based on the test data, a quality measure is determined in step S6. This can be achieved, for example, by comparing the system behavior (system states) with desired system states. For instance, the control deviations can be integrated or accumulated to obtain a quality measure that indicates the robustness of the control system.

[0083] In the optional step S7, the data for creating the quality function is supplemented by the test model parameter set. θ ' and the associated quality measure J( θ ') , i.e. the quality of the resulting control, is supplemented and the Gaussian process model that forms the quality function is retrained or updated.

[0084] In step S8, it is checked whether one of the model parameters of the test model parameter set lies on a boundary of the model parameter domain. If this is the case (alternative: Yes), in step S9 the boundary of the parameter value range on which one of the model parameters lies is extended by an extension measure. Otherwise (alternative: No), the procedure continues with step S10.

[0085] In Figure 3 The diagram schematically depicts the behavior of a real (K1) and a modeled (K2) quality measure as a function of one dimension of the input variables in a one-dimensional case. i The value range of the relevant model parameter within the model parameter domain of the optimization procedure is specified, and point P1 represents the model parameter value that lies at a boundary of the model parameter domain. In this case, the model parameter domain is extended by the augmentation measure ΔΘ. i extended to achieve the optimized model parameter P2, denoted as a triangle, through subsequent iterations of the Bayesian optimization procedure.

[0086] During the Bayesian optimization process, an estimate of the optimum of the model parameter values ​​is available, i.e., the minimum of the approximate quality function in the current model parameter domain Θ. i If Bayesian optimization reveals that the estimated optimum lies at a boundary of the model parameter domain, it is likely that better model parameters lie outside the current model parameter domain. Therefore, it is proposed to define the range of values ​​of the model parameter whose value lies at the boundary of the model parameter domain Θ. i lies in expanding. This dynamic adjustment of the model parameter domain Θ i can be done in various ways.

[0087] The extension measure ΔΘ i may be specified depending on one or more of the following: the gradient of the Gaussian process posterior mean at the point of the ongoing estimation of the optimum ∇ θi µ GP ( θ* ) . The gradient is the derivative of this Gaussian process posterior mean with respect to the relevant model parameter. θ iIf the gradient at the boundary is high, the potentially better value of the model parameter is further from the boundary than if the gradient is low. That is, with a high gradient, the range of values ​​for the model parameter in question is extended by a larger value than with a low gradient. This is related to the length scale of the model parameter that lies on a boundary of the model parameter domain, as measured by the Gaussian process that approximates the quality function. For large length scales, the model assumes that the quality function changes only slowly, and therefore the measure for extending the range of values ​​for the model parameter in question should be increased accordingly. This is related to the extent of the range of values ​​Θ. i of the relevant model parameter. If the range of values ​​is large, the degree of expansion is correspondingly high, and vice versa.

[0088] In step S10, a termination condition is checked, for example, the time to be spent optimizing the quality function, the number of iterations, or a suitable convergence criterion. If the optimization process continues (alternative: No), the process returns to step S4. Otherwise (alternative: Yes), the process is terminated.

[0089] The model parameters of the control strategy can be continuously or at regular or predetermined intervals during the control of the technical system 2.

[0090] The timing will be adjusted so that the control of technical system 2 will gradually improve during operation.

[0091] The following table illustrates exemplary application areas of the above method for evaluating a trajectory: Intended use Regulation to be optimized Machine tools, robots Position control of an electric machine for position setting or motion control of robots drones or autonomous robots Balance control internal combustion engines Air system control, in particular lambda control, injection control

Claims

1. Computer-implemented method provided for creating control for a technical system (2) by means of a Bayesian optimization method, wherein the control is created and can be executed based on model parameters (θ) of a control model, wherein the technical system (2) is a robot, and wherein the following steps are carried out for optimizing the control: - providing a quality function, the form of which corresponds to a trainable regression function and which assesses the quality of control of the technical system (2) based on model parameters (θ); - performing (S4) a Bayesian optimization method based on the quality function in order to iteratively optimize a model parameter set with model parameters (θ) within an initially predefined model parameter domain (Θ); wherein, before performing the Bayesian optimization method, initial control models are created based on a plurality of dynamic system models and the model parameter domain (Θ), which specifies the permissible value ranges for the model parameters (Θ), is determined by determining a distribution of the model parameters (θ) of the initial control models and initially defining the value ranges of the model parameters (θ) depending on an associated maximum a-posteriori estimated value (θMAP) and its respective standard deviation of the distribution of the model parameters (θ).

2. Method according to Claim 1, wherein the dynamic system models are predefined, in particular based on physical model equations.

3. Method according to Claim 1 or 2, wherein the control models (Πθ(x)) are predefined as a linear state control strategy according to Πθ(x) = -K(θ)x, where K represents a control matrix characterizing the control.

4. Method according to Claim 3, wherein the control models are each predefined by a control matrix K=dlqr(A,B,Q,R).

5. Method according to Claim 4, wherein the value ranges for the model parameters (θ) are determined depending on a maximum posterior estimated value (θMAP) of the elements of the control matrix (K).

6. Method for adaptively controlling a technical system (2), wherein model parameters (θ) of the control model are adapted continuously or at regular or predetermined times during the control of the technical system (2) based on the method according to one of Claims 1 to 5.

7. Device for creating control for a technical system (2) by means of a Bayesian optimization method, wherein the control is created and can be executed based on model parameters (θ) of a control model, wherein the technical system (2) is a robot, and wherein the following steps are carried out for optimizing the control: - providing a quality function (J(θ)) which corresponds to a trainable regression function and which assesses the quality of control of the technical system (2) based on model parameters (θ); - performing a Bayesian optimization method based on the quality function (J(θ)) in order to iteratively optimize a model parameter set with model parameters (θ) within an initially predefined model parameter domain (Θ); wherein, before performing the Bayesian optimization method, initial control models are created based on a plurality of dynamic system models and the model parameter domain (Θ), which specifies the permissible value ranges for the model parameters (θ), is determined by determining a distribution of the model parameters (θ) of the initial control models and initially defining the value ranges of the model parameters (θ) depending on an associated maximum a-posteriori estimated value (θMAP) and its respective standard deviation of the distribution of the model parameters (θ).

8. Control system (1) having a technical system (2) and a control unit (3) for controlling the technical system (2), wherein the technical system (2) is a robot, wherein a control model for providing an input variable vector depending on state variables of the technical system (2) is implemented in the control unit (3), wherein a model creation block (4) is provided in order to iteratively optimize a model parameter set with model parameters (θ) for the control model based on a Bayesian optimization method carried out in an optimization block (5) within an initially predefined model parameter domain (Θ), wherein the Bayesian optimization method is performed based on a predefined quality function which is determined or predefined in a quality function block (6), wherein, before performing the Bayesian optimization method, initial control models are created based on a plurality of dynamic system models, and the model parameter domain (Θ), which specifies the permissible value ranges for the model parameters (θ), is determined by determining a distribution of the model parameters (θ) of the initial control models and initially defining the value ranges of the model parameters (θ) depending on an associated maximum a-posteriori estimated value (θMAP) and its respective standard deviation of the distribution of the model parameters (θ).

9. Computer program comprising program code means which is configured to carry out a method according to any of Claims 1 to 6 when the computer program is executed on a computing unit.

10. Machine-readable storage medium with a computer program according to Claim 9 stored thereon.