Method and device for determining a control strategy for a technical system

Abstract

The invention relates to a computer-implemented method for creating a control strategy for a technical system by means of a Bayesian optimization method, wherein the control is created and can be implemented based on model parameters of a control model, wherein the following steps are implemented to optimize the control: - a quality function is provided, the form of which corresponds to a trainable regression function, and the quality function evaluates the quality of the control of the technical system based on the model parameters; - a Bayesian optimization method is performed based on the quality function in order to iteratively optimize a model parameter set with model parameters within an initially predetermined model parameter domain; wherein before the Bayesian optimization method is performed, an initial control model is created based on a plurality of dynamic system models and a model parameter domain is determined, which describes the permissible value range of the model parameters.

Description

Technical Field

[0001] The present invention relates to a method for finding a control strategy suitable for a technical system, in particular using a Bayesian optimization method. The present invention particularly relates to a computer-implemented method for more efficiently optimizing model parameters of a regulation model to realize a control strategy for a technical system. Background Art

[0002] In so-called reinforcement learning, the control system repeatedly interacts with its environment and automatically learns its action strategy from the system's characteristics. The learning process is carried out by minimizing a cost value, described by a cost function or quality function, which evaluates the performance of the control system.

[0003] Model-free reinforcement learning methods are particularly advantageous because they do not require knowledge of the surrounding environment or its interactions with the system. However, the system interacts with the surrounding environment extensively during the learning process. Conventional model-based methods, on the other hand, initially predefine a model structure that essentially describes the behavior of the control system in its environment. This allows the effects of the interaction between the control system and the surrounding environment to be mapped. Adapting the model structure through parameter adaptation is simple and efficient. However, conventional methods have drawbacks in that the selected model structure may be inappropriate, and thus parameter adaptation by optimizing the control model may not lead to optimal results.

[0004] The creation and optimization of control models associated with reinforcement learning can be performed in conjunction with Bayesian optimization, where the cost function is modeled as a Gaussian process model. This enables the creation of efficient black-box optimizers for control models that are otherwise expensive to create / parameterize. However, due to the large number of model parameters in such control models, the optimization effort is high, requiring numerous measurements and resulting in long training times due to the large amount of measurement data. Summary of the Invention

[0005] According to the invention, a method for creating a regulation for a technical system and a corresponding device and system are provided.

[0006] Other embodiments are described in the following description.

[0007] According to a first aspect, a method is provided for creating a regulation for a technical system by means of a Bayesian optimization method, wherein the regulation is created and can be implemented based on model parameters, wherein the following steps are implemented to optimize the regulation:

[0008] - providing a quality function, the form of which corresponds to a trainable regression function and which evaluates the quality of the regulation of the technical system based on the model parameters;

[0009] - performing a Bayesian optimization method based on the quality function to iteratively optimize a model parameter set having model parameters;

[0010] - before executing the Bayesian optimization method, initial control models are created based on multiple dynamic system models and a model parameter range is determined that describes the permissible value range of these model parameters, by determining the distribution of the model parameters of these initial control models and initially defining the value range of these model parameters based on the associated maximum a posteriori estimates and the corresponding standard deviations of the distributions of the model parameters of these initial control models.

[0011] In order to model the regulation of a regulation system, a regulation model is created. The regulation model defines the regulation strategy and represents a functional relationship by which the regulation quantity is calculated based on one or more input variables of the regulation. In this case, at least one of the input variables corresponds to a state variable of the technical system to be regulated.

[0012] The form of the control model can be predetermined in various ways. Thus, the control model may contain or be based on formulaic relationships based on physical laws, such as equations of motion, vibration equations, damping characteristics, and so on. The control model may also be based on freely selected or otherwise determined formulaic relationships. These often have to take into account the nonlinear properties of the technical system.

[0013] Bayesian optimization methods for determining a control model iteratively apply different control strategies to a technical system and efficiently optimize the control. The quality function can be modeled using a trainable regression function, in particular a Gaussian process regression, to model the performance of the system model as a function of the model parameters of the control model, wherein the Gaussian process regression is generated based on noisy (currently measured) state variables.

[0014] In principle, the problem involves finding a control strategy that maps the system state to an input variable vector. To this end, the characteristics of the control of the technical system within a predetermined time range are analyzed in relation to the model parameters of the control strategy based on a state variable vector (composed of a plurality of state variables) and an input variable vector (composed of a plurality of input variables) in order to determine the quality of the characteristics of the technical system. The quality of the characteristics of the technical system can be determined by an evaluation with respect to the desired characteristics. For example, criteria such as oscillation tendency, oscillation duration, interference sensitivity, robustness, etc. can be evaluated. The characteristics of the control of the technical system are also referred to as unknown functions. The quality function (cost function) corresponds to a mathematical model that evaluates the quality of the characteristics of the control of the technical system based on the model parameters of the underlying control strategy.

[0015] In general, Bayesian optimization is used when an unknown function f, a so-called "black-box" function, corresponding to the characteristics of a technical system, is to be minimized. This unknown function f can be analyzed or measured only for values ​​x and (possibly due to noise) observed. The observed value y is y = f(x) + e, where e represents the noise component. It is also assumed that each analysis / measurement of the unknown function f is expensive, i.e., that the analysis of the unknown function incurs high costs, as is the case, for example, when conducting experiments on a bench. Since the analysis of the unknown function is expensive, it is desirable to perform only a small number of analyses during the optimization.

[0016] Under certain preconditions, such as system characteristics or the continuity of the unknown function, the unknown function can be approximated by the mass function using Gaussian process regression. n ) and observation of the corresponding function values ​​(y1, ...yn) allow the quality function to be trained using a Gaussian process. A characteristic of a Gaussian process is that within the range around the analysis point, the model predictions are very good, and the unknown function, in other words, the system characteristics, are well approximated. This is reflected in the low uncertainty of the quality function. Moving away from the analysis point, the model prediction of the quality function for the unknown function deteriorates, and the uncertainty increases with distance from the analysis point.

[0017] A possible strategy for optimizing the model parameters of the control strategy and thereby optimizing the system characteristics is to analyze the unknown function at a number of different positions (e.g., on a regular grid) and assume the lowest observed function value as the optimization result. This approach is not efficient and requires a correspondingly high number of analyses to find the optimal value.

[0018] As an alternative to this approach, a quality function is used to select new analysis points. To this end, the new analysis points for analyzing / measuring the system characteristics are selected so that, on the one hand, the persuasiveness of the quality function is improved, and the uncertainty of the estimated expected value of the quality function is reduced. For this purpose, the analysis points are usually selected within a range in which the unknown function has not yet been analyzed or the technical system has not yet been measured (exploration). On the other hand, the new analysis points are selected for analyzing the unknown function or for measuring the technical system so that the goal of finding the optimal value is achieved as quickly as possible or with a small number of measurements at the analysis points. For this purpose, based on a Gaussian process, analysis points that indicate low function values ​​are preferably selected (exploration). These two conflicting criteria are balanced by selecting the analysis points with the aid of a predetermined acquisition function.

[0019] The acquisition function uses parameters of the quality function described by a Gaussian process model, such as the expected value μ(x) and the standard deviation σ(x) assigned to it. An example is the so-called Lower Confidence Bound (LCB) acquisition function, which is described as follows: LCB(x) = μ(x) - kσ(x). In practice, the factor k is often constant, for example, at a specific value, such as k = 2. This new criterion can be efficiently minimized using conventional gradient-based methods, and the minimum value of LCB(x) then forms the new analysis point for the unknown function f or the measurement of the technical system. It should be noted that in order to optimize the quality function, a model parameter range must be defined in advance. This model parameter range specifies the value range of the corresponding model parameters within which the next analysis point is searched. Previously, these model parameter ranges were usually selected based on experience and / or expert knowledge.

[0020] Based on the analysis of the characteristics of the technical system (e.g., by measurements at the analysis point), the trainable regression function is updated or retrained with the new data, and the next analysis point is selected according to the method described above. Typically, the method is repeated until an abort criterion is met, i.e., the time period that should be used for the optimization of the unknown function f (i.e., the characteristic of the technical system).

[0021] After the optimization method has concluded, the function value x is recommended as the location of the minimum of the quality function, ie, the value which minimizes the expected value of the Gaussian process.

[0022] The LCB criterion is not the only acquisition function proposed in the literature. Other criteria include the expected improvement (EI), the upper confidence bound (UCB), the probability of improvement (PI), or the so-called entropy search method, which is based on information-theoretic considerations.

[0023] According to the above-described method, a Bayesian optimization method is used to find the model parameters of the control strategy by minimizing an initially unknown quality function. The goal is to create a control model so that the characteristics of the technical system correspond to the desired specifications as accurately as possible. The quality function can be continuously updated according to the corresponding model parameters, corresponding to the performance of the control system, as necessary. To update the quality function, the corresponding model parameters need to be evaluated, which requires operating the control system in a real environment with the corresponding model parameters. During the operation of the control system, a quality measure that reflects the quality of the characteristics of the technical system thus regulated is detected and assigned to the current model parameters. Since the control system needs to be operated in real time for measurement processes, improving the cost assessment through the continuously updated quality function is very expensive. Therefore, it is desirable to minimize the number of measurement processes in the real technical system.

[0024] After one or more measurement processes, the quality function with the final measured model parameters and the resulting quality measures can be updated, and a new test model parameter set can be selected by minimizing or maximizing a predetermined acquisition function for evaluating the profile of the quality function or the parameters. The acquisition function is predetermined such that it ensures a compromise between exploration and exploitation. In the context of the method according to the invention, exploration means that a range of the possible model parameter space is preferably used for selecting new test model parameters, within which no quality measures or evaluation system characteristics have yet been determined by previous measurement processes. In the context of the method according to the invention, exploitation means that test model parameters are selected for the next evaluation by the measurement process, which test model parameters are estimated as the optimal model parameter set.

[0025] In the following, it is assumed by way of example that the control strategy corresponds to π θ A linear state control strategy where (x)=-K(θ)x, where K is the control matrix representing the control. This linear control strategy has the following advantages: it is less dimensional than other control models. It also allows for simple implementation in the controller, which improves the efficiency of Bayesian optimization.

[0026] The control strategy is optimized by adapting the elements of the control matrix. The elements of the control matrix can be assumed directly as model parameters or can be assumed to be dependent on the model parameters.

[0027] The idea behind the above-described method is to determine a control model within a restricted model parameter domain using a Bayesian optimization method. While conventional Bayesian optimization methods can be difficult to determine the value ranges of the individual model parameters within which the search for a control strategy should be performed, so that this search can be carried out more efficiently, i.e., using fewer measurement processes, the present method allows for the determination of reasonable value ranges for the model parameters of the control model by initially prescribing a dynamic system model and converting it into an initial control model.

[0028] In this way, the model parameter range for the Bayesian optimization can be automatically defined before the optimization method, ie, without manually predefining the value range of the model parameters, so that the number of measurement processes for the optimization can be significantly reduced.

[0029] In particular, the control model can be predefined by the control matrix K=dlqr(A, B, Q, R). The value ranges of the model parameters can be determined based on maximum a posteriori estimates (MAP estimates) of the elements of the control matrix K. The value ranges can also be predefined based on tolerance ranges.

[0030] The model parameters of the control model can also be adapted continuously or at regular or predetermined points in time during the control of the technical system based on the above-described method.

[0031] According to a further aspect, a device is provided for creating a control for a technical system by means of a Bayesian optimization method, wherein the control is created and can be executed based on model parameters of a control model, wherein the device is designed to execute the following steps to optimize the control:

[0032] - providing a quality function, which corresponds to a trainable regression function and which evaluates the quality of the regulation of the technical system based on model parameters;

[0033] - executing a Bayesian optimization method based on the quality function in order to iteratively optimize a model parameter set having model parameters within an initially predetermined model parameter domain;

[0034] Before executing the Bayesian optimization method, an initial control model is created based on multiple dynamic system models and a model parameter range that describes the permissible value range of the model parameters is determined, by determining the distribution of the model parameters of the initial control model and initially defining the value range of the model parameters based on the associated maximum a posteriori estimates and the corresponding standard deviations of the distribution of the model parameters of the initial control model.

[0035] According to another aspect, a control system is provided, which has a technical system and a control unit for controlling the technical system, wherein a control model is implemented in the control unit for providing an input variable vector depending on state variables of the technical system, wherein a model creation block is provided to iteratively optimize a model parameter set having model parameters for a control model within an initially predetermined model parameter range based on a Bayesian optimization method implemented in an optimization block, wherein the Bayesian optimization method is performed based on a predetermined quality function, which is determined or predetermined in the quality function block, wherein before performing the Bayesian optimization method, initial control models are created based on multiple dynamic system models and a model parameter range is determined that describes the permissible value range of these model parameters, by determining the distribution of the model parameters of these initial control models and initially defining the value range of these model parameters based on the associated maximum a posteriori estimates and the corresponding standard deviations of the distributions of the model parameters of these initial control models. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The embodiments are further described below with reference to the accompanying drawings, wherein:

[0037] Figure 1 A schematic diagram of a regulating system is shown, which comprises a regulating unit and a technical system to be regulated;

[0038] Figure 2 A flow chart is shown for illustrating a method for performing an optimization of a control strategy by means of a Bayesian optimization method;

[0039] Figure 3 A diagram showing a schematic diagram of the operating principle of the dynamic adaptation of the model parameter domain is shown. DETAILED DESCRIPTION

[0040] Figure 1 The schematic diagram of an adaptive control system 1 is shown, which is designed to control a technical system 2. The technical system 2 can be, for example, an internal combustion engine of a motor vehicle or a subsystem of the internal combustion engine. A control unit 3 controls the technical system 2 using a sequence of input variables u as control variables, which result in a specific operating point (system state) of the technical system 2. The input variables u usually include a plurality of input variables, which are combined in an input variable vector Furthermore, for each of these input variables (elements of the input variable vector u), the permissible value range applies. The control of the technical system 2 also results in one or more state variables, which are measured at the input variable vector u to be measured and are represented in the form of the state variable vector x.

[0041] By means of one or more sensors 21 which are part of the technical system 2 , one or more measured variables x1 . . . x D The change process of the corresponding state parameters x1(t)…x D (t), each of which describes the system state x of the technical system 2. D corresponds to the number of state variables. The system state of the technical system 2 is thus detected by means of one or more sensors 21 and transmitted to the control unit 3 as state variables of the state variable vector x.

[0042] The input variable u corresponds to the control quantity of the control unit 3, which is based on the state variable x and the control strategy π θ (x) is determined. The technical system 2 is operated according to the input variable u(t) by means of one or more actuators 22 of the technical system 2. For example, this can control the movement of a robot or a vehicle or control the drive unit or driver assistance system of the vehicle. The input variable u can correspond to the voltage applied to the electromechanical regulator (Stellgeber) as the actuator 22, for example. The actuator 22 is controlled according to one or more input variables u and performs corresponding actions. In this case, the actuator 22 may include a control logic (not necessarily structurally integrated) that determines the control variable used to control the relevant actuator 22 according to the input variable u.

[0043] In one embodiment, the control unit 3 is used to control an internal combustion engine as a technical system. To this end, the throttle position, fuel supply, etc. can be predefined as input variables for a throttle actuator or for an injector actuation device, and corresponding state variables such as speed, load, and engine temperature can be received.

[0044] In one embodiment, the control unit 3 is used to control an at least partially autonomous robot, in particular an at least partially autonomous motor vehicle, as a technical system 2. The sensors 21 may, for example, be one or more video sensors, preferably arranged in the motor vehicle, and / or one or more radar sensors, and / or one or more ultrasound sensors, and / or one or more LiDAR (laser radar) sensors, and / or one or more position sensors (e.g., GPS). Alternatively or in addition, the sensors 21 may also include an information system that determines information about the state of the technical system (motor vehicle), such as a weather information system that determines the current or future state of the weather in the surroundings of the motor vehicle.

[0045] In another embodiment, the control unit 3 is used to control functions in a motor vehicle as a technical system. To this end, the accelerator pedal position, steering interventions in the form of wrist torque or steering position, environmental information (such as the position of surrounding objects), braking interventions, etc. can be predefined as input variables, and corresponding state variables describing the driving behavior of the motor vehicle, such as vehicle speed, cornering behavior, distance to surrounding objects, etc., can be received.

[0046] The control unit 3 can use a plurality of measured variables x1 . . . x D Detecting states or state changes of, for example, an at least partially autonomous robot, such as engine speed, vehicle speed, fuel consumption, engine temperature, longitudinal and / or lateral speed, steering angle, yaw rate, etc. The actuator 22 , which is preferably arranged in a motor vehicle, can be, for example, a brake system, a drive system, or a steering system of the motor vehicle.

[0047] Alternatively, the at least partially autonomous robot may also be another mobile robot (not shown), for example a robot that moves forward by flying, floating, diving or walking. The mobile robot may also be, for example, an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot.

[0048] In yet another alternative, the at least partially autonomous robot can also be a household appliance (not depicted), in particular a washing machine, a stove, an oven, a microwave oven, or a dishwasher. Sensors 21, for example optical sensors, can be used to detect the state of an object being handled by the household appliance, for example, in the case of a washing machine, the state of the laundry in the washing machine. The control unit 3 can then determine the type or state of the object and, by measuring the variables x1 . . . x DTo characterize the type or state of the object. Then, the input parameter can be determined so that the household appliance is controlled according to the determined type or determined state of the object. For example, in the case of a washing machine, the washing machine can be controlled according to the material of the laundry placed therein. Then, the input parameter u(t) can be selected based on the determined material of the laundry.

[0049] In another embodiment, the control unit 3 can be used to control a production machine of a production system (technical system 3) by controlling the actuators 22 that control the production machine via input variables. The production machine 11 can be, for example, a machine for punching, sawing, drilling, milling, turning, and / or cutting.

[0050] Sensor 21 can thus be, for example, an optical sensor that detects characteristics of a finished product. It is possible to control actuators 22 that control the production machine based on the determined characteristics of the finished product so that the production machine performs the subsequent processing steps of the finished product accordingly. It is also possible for sensor 21 to determine characteristics of a finished product processed by the production machine and, based on these characteristics, to adapt the control of the production machine for the next finished product.

[0051] The regulation of the regulation unit 3 follows a regulation strategy. The regulation strategy should be adapted through a dynamic process so that the system characteristics with respect to the quality function are optimized. To this end, an optimization method is implemented that optimizes the model parameters of the regulation model on which the regulation strategy is based so that the performance of the regulated technical system 2 is optimized. To this end, a regulation model (dynamic model) is created in the model creation block 4, which forms the basis for the regulation strategy of the regulation unit 3. The model creation block 4 determines the model parameters of the regulation model based on a Bayesian optimization method implemented in the optimization block 5. This is done based on a predetermined quality function, which is determined or predetermined in the quality function block 6.

[0052] In a further preferred embodiment, the regulation unit 3, the model creation block 4, the optimization block 5 and the quality function block 6 are implemented in a computing unit. The computing unit comprises a control device 2, one or more processors and at least one machine-readable storage medium on which commands are stored which, when executed on the processor, cause the computing unit to carry out the method according to the invention.

[0053] The technical system 2 corresponds to a dynamic system that is optimized by means of the control unit 3 using a suitable control strategy, which should be created using a Bayesian optimization method. The Bayesian optimization method is used to determine the control model by iteratively controlling the technical system 2 using different test model parameter sets during the optimization method and adapting the model parameters based on the resulting state variables. A quality function is modeled using Gaussian process regression or other trainable regression methods, which defines the performance of the control of the technical system 2 as a function of the model parameters.

[0054] The performance of the control is determined according to a predetermined quality criterion (quality measure), which assigns a quality of the control to the resulting state variables of the technical system 2 (in particular, their course) based on a comparison with the desired properties of the technical system 2 controlled by the control model based on the model parameters. These state variables are subject to tolerances, so that the quality function is preferably described by Gaussian process regression.

[0055] In principle, the problem involves finding a vector that maps the system state x to the input parameter u = π θ (x)(where ) adjustment strategy, wherein the model parameters of the adjustment strategy are In the model parameter domain Θ, a quality function J is predetermined over a predetermined time range t=0…T, which is dependent on the model parameters θ that are to be optimized by the optimization method, based on the state vector x and the input variable vector uu:

[0056]

[0057] stx t+1 =f(x t ,π θ (x t ))+v

[0058] in Corresponding to the expected value, c(x t ,u t ) is to attach the input parameter vector u t In the case of state vector x t to give a quality measure of the state, and is a state transition model that describes the dynamics of the technical system 2 and to which a noise parameter v ˜ N (0, ∑ v). The quality measure can be determined by comparing the system characteristic with a desired system characteristic, wherein the quality criterion can specify the robustness, oscillation tendency, oscillation time and quality of the control.

[0059] The Bayesian optimization method is used to find the optimized model parameters θ* of the control strategy by minimizing a quality function (which represents, for example, costs). Alternatively, depending on the meaning of the quality measure, the optimized model parameters θ* of the control strategy can also be determined by maximizing the quality function. The goal is for the control strategy to achieve the best possible control of the technical system 2 using the control unit 3, where "best" refers to minimizing / maximizing (optimizing) the quality measure determined by the quality function from the model parameters and describing the predetermined performance of the overall system consisting of the control unit 3 and the technical system 2.

[0060] The quality function can therefore describe a quality measure that assesses the deviation of the behavior of the real technical system 2 from a predetermined desired performance (ideal system behavior) during the time window t=0 ... T. Therefore, analyzing the quality function requires operating the technical system 2 in a real environment during the measurement process. Due to the need to operate the control system comprising the technical system 2 and the control unit 3 in real life, the evaluation of the quality measure of the behavior of the technical system 2 is very complex, so the number of measurement processes used to evaluate a specific control strategy on the real technical system 2 should be minimized.

[0061] In the following it is assumed that the control strategy corresponds to θ A linear state control strategy of (x)=−K(θ)x, where K corresponds to a control matrix whose elements correspond to model parameters of the control strategy or depend on such model parameters.

[0062] The linear control strategy has the advantage that it is less dimensional than other control models. It also allows it to be easily implemented in the controller and thus increases the efficiency of the Bayesian optimization.

[0063] In conjunction with Bayesian optimization, a linear quadratic regulator (LQR regulator), as known in the field of control strategy development, can be used. In the case of an LQR regulator, the system characteristics and the interaction with the surrounding environment are determined during the measurement process by controlled operation with a variable set of input variable vectors and the detection of the resulting state variable vectors. In this case, the system dynamics is determined according to f(x t ,u t )≈Ax t +But

[0064] to linearize and the cost is squared accordingly

[0065]

[0066] By means of these approximations, an LQR feedback matrix can be created in the model creation block 4, which represents a dynamic model and is generally referred to as the control matrix K=dlqr(A, B, Q, R). The control strategy optimization is performed by directly adapting the elements of the control matrix as the model parameters to be optimized (K-learning), wherein a portion or each entry (element) of the entries of the control matrix corresponds to the model parameter to be optimized. Alternatively, only the components of the control matrix K corresponding to the system matrices A and B can be assumed as the model parameters to be optimized, wherein each entry of the matrices A and B corresponds to a model parameter (AB-learning).

[0067] Alternatively, only the matrices Q and R (weighting matrix) can be assumed as model parameters to be optimized in the control matrix K (QR-learning). In this case, it is sufficient to adapt only the diagonal entries of the weighting matrix in the following form:

[0068] K QR (θ)=dlqr(A, B, Q(θ), R(θ)),

[0069] in

[0070] Depending on the ownership of the model parameters to be optimized, these methods are called K learning, AB learning and QR learning.

[0071] In Bayesian optimization, the model parameter domain must be defined before optimization begins in order to find good estimates for the optimized model parameters. However, without prior knowledge, it is difficult to select the value range of the individual model parameters for optimization, i.e., the model parameter domain. However, this is crucial for an efficient optimization method that does not require an excessive number of measurements.

[0072] The method thus provides for selecting a search range suitable for the model parameters so that the number of measurement processes can be reduced. This is particularly necessary in the case of high dimensions, since in such cases it can be difficult to manually set the value range of each of the model parameters. The value ranges of the model parameters are therefore first determined by learning a distribution about the dynamic model and then using this distribution to select the value range of each of the model parameters. This distribution is obtained by Bayesian linear regression from the recorded data values ​​of the state variable vector and the input variable vector in order to obtain an approximate linear model of the system dynamics. This results in a Gaussian distribution about the model.

[0073] p(vec(A,B)|Data)=N(vec(A,B)|μt AB ,∑ AB )

[0074] where μ AB is the maximum a posteriori (MAP) estimate and where the notation vec(.,.) indicates that the matrices A and B are transformed into vectors.

[0075] By determining the control matrix K=dlqr(A, B, Q, R) for a plurality of predefined dynamic system models, the distribution of the model parameters of the elements of the control matrix K can be determined. These distributions are used to select reasonable value ranges for these model parameters. In order to obtain the distribution of possible control matrices, N is calculated using the above-mentioned dlqr function. S pair (A,B) in order to obtain the sample distribution (posterior distribution) Next, the distribution of values ​​is described by the product of independent normal distributions:

[0076]

[0077] With this approximation, it is possible to specify a range Θ of model parameters around the mean of the distribution with a width specified by the standard deviation of the distribution. K :

[0078]

[0079] In this case, the scaling parameter β determines the effective size of the value range of the model parameters.

[0080] As the scaling parameter β increases, the Bayesian optimization method becomes more flexible, allowing for better solutions. However, more measurements are required to determine the model parameters. Therefore, a trade-off must be struck between limiting the range of values ​​for the model parameters to a suitable value and ensuring that the range is large enough to avoid errors caused by incorrect adaptation of the system model. For example, the range of values ​​for the model parameters can be dynamically adapted during optimization.

[0081] To exploit the potential correlation between these model parameters, a multivariate Gaussian process can be used to approximate the value distribution Now, it is proposed here to transform these model parameters into the eigenspace of the covariance matrix ∑K, where The transformation matrix T is composed of eigenvectors ∑K. Then, in this eigenspace, the range of these model parameters is transformed into in The i-th eigenvalue of ΣK is shown. Basically, the principled component analysis PCA is performed.

[0082] After selecting the value ranges for the model parameters, it is possible that the value ranges for these model parameters are chosen too conservatively in certain dimensions. This can occur, for example, when the scaling parameter β is too small or due to model bias. Consequently, during the optimization process, it may occur that the optimal model parameter vector does not lie within the selected model parameter range. Therefore, the model parameter range can be dynamically adapted during the optimization process.

[0083] During the Bayesian optimization process, an estimate of the optimal value of the model parameter domain is obtained, that is, the minimum value of the approximated quality function within the current model parameter domain. If Bayesian optimization indicates that the estimated optimal value is at the boundary of the model parameter domain, then there may be better model parameters outside the current model parameter domain. Therefore, it is proposed to expand the value range of the model parameter whose value is at the boundary of the model parameter domain. This dynamic adaptation of the model parameter domain can be implemented in various ways.

[0084] Adapting the value range of the model parameters allows, starting from a restricted value range, for the dynamic adaptation of the value range during optimization only for those dimensions of the model parameter vector where the optimization involves the boundaries of the model parameter range. This allows for a more efficient optimization overall, significantly improving convergence. Potential model errors can also be better compensated, resulting in a more robust optimized system model. This improved efficiency makes it possible to scale Bayesian optimization to high-dimensional control strategies.

[0085] In order to clarify the creation and adaptation of the regulation of the technical system, the following Figure 2 The method is described in the flowchart.

[0086] For this purpose, a series of system models are predefined in step S1 , which are each based on a physical model equation and determine the control matrix K=dlqr(A, B, Q, R) according to the above-described method. The system models preferably have different parameterizations.

[0087] The distribution of the initial tuning model is obtained by creating a Gaussian distribution for each of the model parameters in the control matrix K

[0088] p(vec(A,B)|Data)=N(vec(R,B)|μ AB ∑ AB )

[0089] where μ AB is the maximum a posteriori (MAP) estimate (the expected value of the corresponding Gaussian distribution) and the notation vec(.,.) indicates that the matrices A and B are transformed into vectors.

[0090] In step S2, according to the element θ of the adjustment matrix K MAP The maximum a posteriori estimate (MAP estimate) is used to determine the distribution. The elements θ of the adjustment matrix K are MAP The maximum a posteriori estimate (MAP estimate) corresponds to

[0091] where n corresponds to the number of data points, γ corresponds to a predetermined constant describing the noise in the data, m0 corresponds to the prior mean of the model parameters to be estimated, S0 corresponds to the prior covariance of the model parameters to be estimated, X corresponds to the position of the data points (grouped in the matrix), t corresponds to the function value of the data points (grouped in the vector), and m n corresponds to the posterior expected value (average value) of the model parameters to be estimated and S n The posterior covariance corresponding to the model parameters to be estimated.

[0092] Next, the distribution of values ​​is described by the product of independent normal distributions:

[0093]

[0094] where n s corresponds to the number of samples, which are determined by p(vec(K)|A, B), and n θ corresponds to the number of parameters that should be estimated.

[0095] With this approximation, it is possible to specify a range Θ of model parameters around the mean of the distribution with a width specified by the standard deviation of the distribution. K :

[0096]

[0097] In this case, the scaling parameter β determines the effective size of the value range of the model parameters.

[0098] As the scaling parameter β increases, the Bayesian optimization method becomes more flexible, allowing for better solutions. However, more measurements are required to determine the model parameters. Therefore, a trade-off must be struck between limiting the range of values ​​for the model parameters to a suitable value and ensuring that the range is large enough to avoid errors caused by incorrect adaptation of the system model. For example, the range of values ​​for the model parameters can be dynamically adapted during optimization.

[0099] To exploit the potential correlation between these model parameters, a multivariate Gaussian process can alternatively be used to approximate the value distribution. Now, it is proposed here to transform these model parameters into the eigenspace of the covariance matrix ∑N, where The transformation matrix T is composed of the eigenvector ∑N. Then, in this eigenspace, the range of these model parameters is transformed into in The i-th eigenvalue of ΣK is shown. Basically, the principled component analysis PCA is performed.

[0100] In step S3, the value ranges of the control parameters to be determined for the control model are specified. Bayesian optimization must also cover the value range of the control parameters as fully as possible, given the length scale of the cost function. It is often difficult to determine a priori the value range within which the control parameters, that is, the entries of the control matrix K, must be optimized to achieve good control characteristics.

[0101] To this end, we use the MAP estimate θ MAP Based on the system identification of step S2 , a reasonable value range is selected for each of the control parameters, and these control parameters are subsequently optimized within this value range.

[0102] In step S4, the optimization process starts.

[0103] To this end, a new test model parameter set θ' is selected, in which D n+1 To make the pre-given acquisition function a(θ; D n ) is maximized or minimized. The acquisition function α(θ; D n) is predetermined such that the acquisition function uses a combination of exploration and mining in order to determine a new set of test model parameters at the minimum or maximum of the acquisition function (depending on the selected acquisition function). The acquisition function uses parameters that are created as a quality function of the Gaussian process model, such as the Gaussian process mean (expected value) and the Gaussian process standard deviation. In the context of the method according to the invention, exploration means that a range of the possible model parameter space is preferably used for selecting new test model parameters, within which the characteristics of the technical system have not yet been evaluated by a previous measurement process. In the context of the method according to the invention, mining means that a set of model parameters is selected for the next evaluation of the characteristics of the technical system by a measurement process, which set of model parameters is estimated to be the best set of model parameters.

[0104] θ * =arg min θ J(θ);θ′←θ * .

[0105] The acquisition function uses parameters of the quality function described by a Gaussian process model, such as the expected value μ(x) and the standard deviation σ(x) assigned to it. An example is the so-called Lower-Confidence-Bound (LCB) acquisition function, which is described as follows: LCB(x) = μ(x) - kσ(x). This new criterion can be efficiently minimized using common gradient-based methods, and the minimum value of LCB(x) then forms a new analysis point for the unknown function f or the measurement of the technical system 2.

[0106] The LCB criterion is not the only acquisition function proposed in the literature. Other criteria include the expected improvement (EI), the upper confidence bound (UCB), the probability of improvement (PI), or the so-called entropy search method, which is based on information-theoretic considerations.

[0107] In step S5 , a measuring process is carried out and the control of the technical system 2 is evaluated by the control unit 3 during the time window t=0 . . . T based on the test model parameter set in order to correspondingly detect test data in the form of input variables u and resulting state variables x.

[0108] In step S6, a quality measure is determined based on the test data. This can be achieved, for example, by comparing the system properties (system state) with the desired system state. For example, the control deviation of the control can be numerically integrated or accumulated to obtain a quality measure that indicates the robustness of the control.

[0109] In optional step S7 , the data for creating the quality function are supplemented with a test model parameter set θ′ and the associated quality measure J(θ′), ie the quality of the resulting control, and the Gaussian process model forming the quality function is retrained or updated.

[0110] In step S8, a check is performed to determine whether one of the model parameters of the test model parameter set is located at a boundary of the model parameter range. If so (alternative: yes), then in step S9, the boundary of the parameter value range at which one of the model parameters is located is expanded by an expansion dimension (β). Otherwise (alternative: no), the method continues with step S10.

[0111] exist Figure 3 In FIG. 1 , a diagram showing the course of the actual quality measure (K1) and the modeled quality measure (K2) as a function of the dimension of the input variables is schematically shown for the one-dimensional case. Θ i The model parameter domain of the optimization method is described as the value range of the relevant model parameter, and the point P1 describes the model parameter value at the boundary of the model parameter domain. In this case, the model parameter domain is extended by the extension size ΔΘ i , so that the optimized model parameters P2 (represented as triangles) are achieved through subsequent iterations of the Bayesian optimization method.

[0112] During the Bayesian optimization process, there are estimates of the optimal values ​​of the model parameter values, that is, the values ​​in the current model parameter domain Θ i The minimum value of the quality function approximated within the model parameter domain. If the Bayesian optimization shows that the estimated optimal value is at the boundary of the model parameter domain, then there may be better model parameters outside the current model parameter domain. Therefore, it is proposed to put its value in the model parameter domain Θ i The range expansion of the model parameter at the boundary of Θ can be implemented in different ways. i This dynamic adaptation.

[0113] Extension size ΔΘ i It can be described according to one or more of the following descriptions:

[0114] -The gradient of the Gaussian process posterior mean at the point where the optimal value is being estimated The gradient is the posterior mean of the Gaussian process according to the relevant model parameters θ i If the gradient is high at the boundary region, then potentially better values ​​for the model parameters are further away from the boundary region than when the gradient is small. That is, when the gradient is high, the value range of the relevant model parameters is extended to higher values ​​than when the gradient is small.

[0115] The length scale of the model parameter at the limit of the model parameter range relative to the Gaussian process approximating the mass function. For large length scales, the model assumes that the mass function changes only slowly and therefore the degree of expansion of the value range of the relevant model parameter should be increased accordingly.

[0116] - the value range of the relevant model parameters Θ i If the value range is large, the degree of expansion is correspondingly high, and vice versa.

[0117] Then, in step S10, the interruption condition is checked, i.e., for example, the time duration or number of iterations or a suitable convergence criterion to be used for optimizing the quality function, and if the optimization method is to be continued (alternative: No), a jump back to step S4 is performed. Otherwise (alternative: Yes), the method is terminated.

[0118] The model parameters of the control strategy can be adapted continuously or at regular or predetermined points in time during the control of the technical system 2 , so that the control of the technical system 2 is significantly improved during operation.

[0119] The following table illustrates exemplary areas of application of the above-described method for evaluating trajectories:

[0120]

Claims

1. A computer-implemented method for creating a regulation for a technical system (2) by means of a Bayesian optimization method, wherein the regulation is created and can be implemented based on model parameters (θ) of a regulation model, wherein the following steps are performed to optimize the regulation: - providing a quality function, the form of which corresponds to a trainable regression function and which evaluates the quality of the regulation of the technical system (2) based on the model parameters (θ); - executing (S4) a Bayesian optimization method based on the quality function in order to iteratively optimize a model parameter set having model parameters (θ) within an initially predetermined model parameter domain (Θ); In this case, before the Bayesian optimization method is carried out, an initial control model is created based on a plurality of dynamic system models and a model parameter range (θ) is determined which describes the permissible value range of the model parameters (θ), by determining the distribution of the model parameters (θ) of the initial control model and calculating the distribution of the model parameters (θ) according to the associated maximum a posteriori estimates (θ). MAP ) and the corresponding standard deviation of the distribution of the model parameter (θ) of the initial adjustment model to initially define the value range of the model parameter (θ).

2. The method according to claim 1, The dynamic system model is predetermined in this case.

3. The method according to claim 2, The dynamic system model is predetermined here on the basis of physical model equations.

4. The method according to any one of claims 1 to 3, According to π θ (x) = -K(θ)x The control model (π θ (x)) as a linear state regulation strategy, where K is a regulation matrix characterizing the regulation.

5. The method according to claim 4, The control models are each predetermined by a control matrix K=dlqr(A, B, Q, R), wherein A, B, Q and R are each matrices.

6. The method according to claim 5, The range of the model parameter (θ) is determined based on the maximum a posteriori estimate (θ) of the elements of the adjustment matrix (K). MAP ) to determine.

7. A method for adaptively regulating a technical system (2), wherein the model parameters (θ) of the regulation model are adapted continuously or at regular or predetermined time points during the regulation of the technical system (2) based on the method according to any one of claims 1 to 6.

8. A device for creating a regulation for a technical system (2) by means of a Bayesian optimization method, wherein the regulation is created and can be implemented based on model parameters (θ) of a regulation model, wherein the following steps are implemented to optimize the regulation: - providing a quality function (J(θ)), which corresponds to a trainable regression function and which evaluates the quality of the regulation for the technical system (2) based on the model parameters (θ); - performing a Bayesian optimization method based on the quality function (J(θ)) in order to iteratively optimize a model parameter set having model parameters (θ) within an initially predetermined model parameter domain (θ); In this case, before the Bayesian optimization method is carried out, an initial control model is created based on a plurality of dynamic system models and a model parameter range (θ) is determined which describes the permissible value range of the model parameters (θ), by determining the distribution of the model parameters (θ) of the initial control model and calculating the distribution of the model parameters (θ) according to the associated maximum a posteriori estimates (θ). MAP ) and the corresponding standard deviation of the distribution of the model parameter (θ) of the initial adjustment model to initially define the value range of the model parameter (θ).

9. A control system (1) comprising a technical system (2) and a control unit (3) for controlling the technical system (2), wherein a control model is implemented in the control unit (3) for providing an input variable vector as a function of state variables of the technical system (2), wherein a model creation block (4) is provided in order to iteratively optimize a model parameter set having model parameters (θ) for the control model within an initially predetermined model parameter domain (θ) based on a Bayesian optimization method implemented in an optimization block (5), wherein the Bayesian optimization method is performed based on a predetermined quality function, which is determined or predetermined in a quality function block (6), wherein before the execution of the Bayesian optimization method, an initial control model is created based on a plurality of dynamic system models and a model parameter domain (θ) is determined which describes an admissible value range of the model parameters (θ), in such a way that a distribution of the model parameters (θ) of the initial control model is determined and a maximum a posteriori estimate (θ) of the model parameters (θ) is calculated based on the associated maximum a posteriori estimate (θ) MAP ) and the corresponding standard deviation of the distribution of the model parameter (θ) of the initial adjustment model to initially define the value range of the model parameter (θ). 10 . Computer program product comprising a computer program with program code means, the computer program being designed to carry out the method according to claim 1 when the computer program is executed on a computing unit. 11 . A machine-readable storage medium having a computer program with program code means stored thereon, the computer program being designed to carry out the method according to claim 1 when the computer program is executed on a computing unit.

Images