Method and device for determining model parameters of a regulation strategy for a technical system

Through Bayesian optimization and Gaussian process regression combined with LQR regulator and AB learning method, dynamically adapting the model parameter range, solving the problems of complexity and low optimization efficiency of regulation model parameters in nonlinear systems, achieving more efficient regulation The strategy is determined.

CN113874865BActive Publication Date: 2024-11-26ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080041182.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-06-06
Filing Date
2020-05-27
Publication Date
2024-11-26
Estimated Expiration
2040-05-27

AI Technical Summary

Technical Problem

In nonlinear physical systems, the parameterization process of creating adjustment models based on physical interrelationships is complex and expensive. The unsuitable model structure selection of traditional methods leads to poor optimization results. The Bayesian optimization method has complex optimization due to the large number of model parameters. It requires a lot of measurement process and a long training time.

Method used

The Bayesian optimization method is used to determine the adjustment model parameters through iterative quality function, and the Gaussian process regression modeling system performance is used, combined with the LQR regulator and AB learning method, and dynamically adapt the model parameter range to reduce the measurement process and optimize the adjustment strategy.

Benefits of technology

The number of measurement processes required to optimize the model parameter is significantly reduced, the optimization efficiency is improved, the training time is shortened, and more accurate adjustment strategy determination is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113874865B_ABST
    Figure CN113874865B_ABST
Patent Text Reader

Abstract

The invention relates to a method for determining a control strategy for a technical system (2) by means of a Bayesian optimization method, wherein the control strategy is created and can be implemented based on model parameters (#imgabs0#) of a control model, wherein the following steps are performed for optimizing the control: a quality function is provided, the form of which corresponds to a regression function and which evaluates the control quality of the technical system (2) based on the model parameters (#imgabs1#); a Bayesian optimization method is performed based on the quality function in order to iteratively determine a model parameter set with model parameters (#imgabs3#) within a model parameter domain (I) that specifies the permissible value range of the model parameters (#imgabs2#); and a model parameter domain (I) of at least one of the model parameters is determined based on an associated maximum a posteriori estimate (#imgabs4#) of the quality function.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for finding a suitable control strategy for a technical system, in particular using a Bayesian optimization method. In particular, the invention relates to a computer-implemented method for more effectively optimizing model parameters of a control model in order to implement a control strategy for a technical system. Furthermore, the invention relates to measures for accelerating the implementation of the Bayesian optimization method in order to reduce the computational effort of parameterizing the control strategy. Background Art

[0002] The determination of a control strategy or control model for regulating or operating a technical system is complex, depending on the complexity of the technical system. Although the optimal control can be determined by classical methods in the case of linear dynamic behavior of the technical system, in particular in the case of nonlinear physical systems, it is complex to create or parameterize a corresponding control model based on physical relationships.

[0003] Alternatively, so-called reinforcement learning can be used to find a suitable control strategy. Reinforcement learning involves observing the interaction between the technical system and the environment and learning a corresponding function model therefrom, which describes the behavior of the technical system when it interacts with the environment. The learning process is carried out by minimizing a cost value described by a cost function or quality function, which evaluates the performance of the control system to obtain model parameters of a control model for automatically and controllably guiding the technical system in the environment.

[0004] In particular, model-free reinforcement learning methods are advantageous because no knowledge about the environment and the interaction between the environment and the system is required, but the interaction time between the system and the environment is very long during the learning process. In contrast, in the case of conventional model-based methods, a model structure is initially predetermined, which essentially describes the behavior of the control system in the environment. This allows the influence of the interaction between the control system and the environment to be mapped. Adapting the model structure by parameter adaptation can be performed simply and efficiently. However, conventional methods have disadvantages because the selected model structure may not be suitable, and thus parameter adaptation by optimization of the control model does not lead to optimal results.

[0005] The creation and optimization of regulation models associated with reinforcement learning can be performed in conjunction with Bayesian optimization, where the quality function is modeled as a Gaussian process model. This enables the creation of an efficient black-box optimizer for regulation models that would otherwise be very complex to create / parameterize. However, due to the large number of model parameters of such regulation models, the optimization is very complex, so that extensive measurement processes are necessary and, due to the large amount of measurement data, long training times are the rule. Summary of the invention

[0006] According to the invention, a method for determining a control strategy for a technical system by means of a Bayesian optimization method as well as a corresponding device, a control system and a control method are provided.

[0007] Further embodiments are described in the present disclosure.

[0008] According to a first aspect, a method for determining a control strategy for a technical system by means of a Bayesian optimization method is provided, wherein the control strategy is created and can be implemented based on model parameters of a control model, wherein the following steps are performed for optimal control:

[0009] - providing a quality function, the form of which corresponds to a regression function and which evaluates the regulation quality of the technical system based on the model parameters;

[0010] - performing a Bayesian optimization method based on the quality function in order to iteratively determine a model parameter set having model parameters within a model parameter domain describing a range of permissible values ​​for the model parameters; and

[0011] + Determining a model parameter range for at least one of the model parameters as a function of the associated maximum a posteriori estimate of the quality function.

[0012] A control model is created to model the control of a control system. The control model defines a control strategy and represents a functional relationship with which a manipulated variable can be calculated based on one or more input variables of the control. At least one of the input variables corresponds to a state variable of the technical system to be controlled.

[0013] The form of the control model can be predetermined in various ways. Thus, the control model can contain formula relationships based on physical laws, such as equations of motion, vibration equations, damping behavior, etc. The control model can also be based on freely selected formula relationships. The formula relationships usually have to take into account the nonlinear behavior of the technical system.

[0014] The Bayesian optimization method and technology for determining a control model systematically applies different control strategies iteratively and optimizes the control in an efficient manner. In this case, a quality function is modeled with the aid of a Gaussian process regression in order to model the performance of the system model as a function of the model parameters of the control model, wherein the Gaussian process regression is created based on noisy state variables. Basically, the problem involves finding a control strategy that maps the system state onto an input variable vector. For this purpose, a quality function (cost function) that depends on the model parameters of the control strategy is evaluated within a predetermined time frame based on a state variable vector (from a plurality of state variables) and an input variable vector (from a plurality of input variables).

[0015] Bayesian optimization is generally used when an unknown function f, a so-called "black box" function, is to be minimized. This unknown function can only be evaluated and observed (possibly with noise) for values ​​x. The observed value y is obtained as y=f(x)+e, where e represents the noise. It is also assumed that each evaluation of the unknown function f is expensive, that is to say leads to costs in the sense that the evaluation of the unknown function leads to high expenditure, which is the case, for example, when carrying out experiments on a test bench. Since the evaluation of the unknown function is expensive, it is desirable to carry out only a small number of evaluations during the optimization.

[0016] Under certain assumptions, such as the continuity of the unknown function, Gaussian process regression can be used to approximate the unknown function in the function model. To this end, at multiple evaluation points (X 1 , …, x n ) and use the Gaussian process to observe the corresponding function value (y 1 ,…y n ) After that, a model of the unknown function f can be built. One property of the Gaussian process is that the model predictions are very good in the range around the evaluation point and can approximate the unknown function well. This is reflected in the low uncertainty of the function model. Away from the evaluation point, the predictions about the unknown function f become worse, and the uncertainty increases with the distance from the evaluation point.

[0017] One possible strategy for optimizing the unknown function f is to evaluate the unknown function at many different locations (e.g., on a regular grid) and take the lowest observed function value as the result of the optimization. This approach is inefficient and requires many evaluations with correspondingly high costs to find the optimal value.

[0018] Instead of this solution, a Gaussian process is used to select new evaluation points. For this purpose, the new evaluation points for evaluating the unknown function are selected so that, on the one hand, the model is improved, thereby reducing the uncertainty of the estimated expected value of the quality function. For this purpose, the evaluation points are usually selected in areas where the unknown function has not yet been evaluated (Exploration). On the other hand, the new evaluation points for evaluating the unknown function are selected so that the goal of minimizing the function is achieved as quickly as possible or with a small number of measurements at the evaluation points. For this purpose, evaluation points that promise low function values ​​based on Gaussian processes (Exploitation) are preferred. These two opposing criteria are weighed by the so-called acquisition function.

[0019] The acquisition function uses the parameters of the quality function described by the Gaussian process model, such as the expected value μ(x) and the standard deviation σ(x) assigned to the expected value. An example is the so-called Lower Confidence Bound (LCB) acquisition function, which is described as follows: LCB(x)=μ(x)-kσ(x). In practice, the factor k is usually constant, for example specified as a specific value, such as k=2. This new criterion can be effectively minimized using common gradient-based methods, and the minimum value of LCB(x) then forms a new evaluation point of the unknown function. It should be noted in this case that for the optimization of the acquisition function, an optimization domain must be defined in which the next evaluation point is searched. This domain is usually selected based on experience and / or expertise.

[0020] According to the above method, the Bayesian optimization method is used to find the model parameters of the regulation strategy by minimizing the initially unknown quality function. The goal is to create a regulation model in such a way that the behavior of the technical system meets the desired specifications as accurately as possible. If necessary, the quality function can be continuously updated according to the corresponding model parameters based on the performance of the regulation system. In order to update the quality function, the corresponding model parameters need to be evaluated, which makes it necessary to operate the regulation system with the corresponding model parameters in a real environment. Due to the need to actually operate the regulation system, it becomes very expensive to improve the cost assessment by the quality function that is constantly updated. Therefore, it is desirable to minimize the number of measurement processes at the actual technical system as much as possible.

[0021] After one or more measurement processes, respectively, the quality function can be updated with the most recently measured model parameters and the resulting costs, and a new set of test model parameters can be selected in which a predetermined acquisition function for evaluating the parameters of the quality function is minimized or maximized. The acquisition function is predetermined so that a new set of test model parameters is determined by minimization / maximization with a trade-off between exploration and adoption. In the context of the method according to the invention, exploration means that for the selection of new test model parameters, the following regions of the possible model parameter space are preferred in which the costs have not yet been evaluated by a previous measurement process. Adoption in the context of the method according to the invention means selecting a model parameter set for the next cost evaluation by means of a measurement process, which model parameter set is estimated to be an optimal set of model parameters.

[0022] The idea of ​​the method described above is to determine the model parameters of the control model in a previously restricted optimization domain with the aid of a Bayesian optimization method. Although it is not easy to determine the model parameter domain of the individual model parameters in which the search for the control strategy is to be carried out in the case of conventional Bayesian optimization methods so that the optimization is carried out in a more efficient manner, i.e. with a smaller number of evaluation processes / measurements, the value ranges of the model parameters for their optimization can be determined based on previously determined maximum a posteriori estimates (MAP estimates) from the initially carried out system identification.

[0023] In this way, the model parameter domain for the Bayesian optimization of the quality function can be automatically narrowed, ie, the value range of the model parameters does not need to be manually predefined, so that the number of measurement processes for the optimization can be significantly reduced.

[0024] Furthermore, the parametric regression model can map the input variable vector and the system state onto the subsequent system state and can in particular be trained accordingly to obtain a weighting matrix.

[0025] Furthermore, the model parameters can be determined by means of an AB learning method for an LQR controller, wherein an uncertainty measure is respectively determined for at least one maximum a posteriori estimate, wherein a value range of at least one model parameter is defined around the maximum a posteriori estimate.

[0026] In addition to the maximum a posteriori estimate, an uncertainty measure for the maximum a posteriori estimate can also be determined and allows the size of the corresponding value range of the assigned model parameter to be determined. This enables the optimized search range of the model parameters to be adapted individually, i.e. the search range for model parameters that are subject to a high degree of uncertainty is specified to be larger than for model parameters that are subject to a lower degree of uncertainty.

[0027] It may be provided that the model parameters are determined by means of a K-learning method for the LQR controller, wherein a value range of at least one model parameter is defined around a maximum a posteriori estimate.

[0028] Furthermore, a value range of at least one model parameter is determined around the maximum a posteriori estimate using a metric that is determined as a product of a predefined factor between 0 and 1 and the associated maximum a posteriori estimate.

[0029] According to another embodiment, the optimization method can start with initial model parameters obtained by minimizing an a priori mean value function, wherein the non-parametric approximation model of the technical system is trained to obtain the a priori mean value function.

[0030] According to a further aspect, a device is provided for determining a control strategy for a technical system by means of a Bayesian optimization method, wherein the control strategy is created and can be implemented based on model parameters of a control model, wherein the device is designed to carry out the following steps for optimized control:

[0031] - providing a quality function, the form of which corresponds to a regression function and which evaluates the regulation quality of the technical system based on the model parameters;

[0032] - performing a Bayesian optimization method based on the quality function in order to iteratively determine a model parameter set having model parameters within a model parameter domain that specifies a range of permissible values ​​for the model parameters;

[0033] - determining a model parameter range for at least one of the model parameters as a function of the associated maximum a posteriori estimate of the quality function.

[0034] According to a further aspect, a control system is provided having a technical system and a control unit for controlling the technical system, wherein a control model for providing an input variable vector as a function of state variables of the technical system is implemented in the control unit, wherein a model creation block is provided for determining model parameters of the control model based on a Bayesian optimization method implemented in an optimization block, wherein a control strategy is created and can be implemented based on the model parameters of the control model, wherein the following steps are implemented for optimizing the control:

[0035] + providing a quality function, the form of which corresponds to the regression function and which evaluates the regulation quality of the technical system based on the model parameters;

[0036] + performing a Bayesian optimization method based on the quality function in order to iteratively determine a model parameter set having model parameters within a model parameter domain that describes a range of permissible values ​​for the model parameters;

[0037] - determining a model parameter range for at least one of the model parameters as a function of the associated maximum a posteriori estimate of the quality function. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The following is a more detailed explanation of the implementation mode based on the accompanying drawings, wherein:

[0039] Figure 1 shows a schematic diagram of a control system having a control unit and a technical system to be controlled; and

[0040] Figure 2 is a flowchart that illustrates the method of creating a conditioning model using reinforcement learning methods. DETAILED DESCRIPTION

[0041] Figure 1The schematic diagram of an adaptive control system 1 designed to control a technical system 2 is shown. The technical system 2 can be, for example, an internal combustion engine of a motor vehicle or a subsystem thereof. A control unit 3 controls the technical system 2 using a series of input variables u as manipulated variables, which lead to certain operating points of the technical system 2. The input variables u usually include a plurality of input variables, which are combined in an input variable vector In addition, there is a permissible value range for each of the input variables (elements of the input variable vector u). Furthermore, the control of the technical system 2 results in one or more state variables, which are measured at the input variable vector u to be measured and are represented in the form of a state variable vector x.

[0042] With the aid of one or more sensors 21 as part of the technical system 2, one or more measured variables x can be detected. 1 …x D The one or more measured parameters represent the corresponding state parameters x 1 (t)...x o (t), the state variables respectively describe the system state x of the technical system 2. D corresponds to the number of state variables. The system state of the technical system 2 is thus detected by means of one or more sensors 21 and transmitted to the control unit 3 as state variables of the state variable vector x.

[0043] The input parameter u corresponds to the state parameter x and the regulation strategy π θ (x) Determined control variables of the regulating unit 3. The operation of the technical system 2 is carried out with the aid of one or more actuators 22 of the technical system 2 as a function of the input variable u(t). For example, the movement of a robot or a vehicle can be controlled or a drive unit or a driver assistance system of the vehicle can be controlled. For example, the input variable u can correspond to a voltage applied to an electromechanical actuator as an actuator 22. The actuator 22 is controlled as a function of one or more input variables u and a corresponding action is performed. In this case, the actuator 22 can include a control logic (not necessarily structurally integrated) which determines the control variable from the input variable u, with which the relevant actuator 22 is controlled.

[0044] In one embodiment, the control unit 3 is used to control an internal combustion engine as a technical system. For this purpose, the throttle position, fuel delivery and / or the like can be predefined as input variables to a throttle actuator or an injection valve control device and corresponding state variables such as speed, load, engine temperature can be received.

[0045] In one embodiment, the control unit 3 is used to control an at least partially autonomous robot, in particular an at least partially autonomous motor vehicle, as a technical system 2. The sensor 21 can be, for example, one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR (laser radar) sensors and / or one or more position sensors (e.g. GPS), which are preferably arranged in the motor vehicle. Alternatively or additionally, the sensor 21 can also include an information system that determines information about the state of the technical system (motor vehicle), for example a weather information system that determines current or future weather conditions in the vicinity of the motor vehicle.

[0046] In another exemplary embodiment, the control unit 3 is used to control functions in a motor vehicle as a technical system. For this purpose, the accelerator pedal position, steering interventions in the form of wrist torque or steering position, environmental information such as the position of surrounding objects, braking interventions and / or the like can be predefined as input variables and corresponding state variables describing the driving behavior of the motor vehicle, such as vehicle speed, cornering position, distance to surrounding objects, etc., can be received.

[0047] The control unit 3 can use a plurality of measured variables x1 . . . x D To detect, for example, a state or a state change process of the at least partially autonomous robot, such as engine speed, vehicle speed, fuel consumption, engine temperature, longitudinal speed and / or lateral speed, steering angle, yaw rate, etc. The actuator 32, which is preferably arranged in the motor vehicle, can be, for example, a brake, a drive or a steering gear of the motor vehicle.

[0048] Alternatively, the at least partially autonomous robot may also be another mobile robot (not shown), for example a robot that moves by flying, swimming, diving or walking. The mobile robot may also be, for example, an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot.

[0049] In a further alternative, the at least partially autonomous robot can also be a household appliance (not shown), in particular a washing machine, a stove, an oven, a microwave or a dishwasher. The state of an object handled by the household appliance, for example the state of the laundry in the washing machine, can be detected by means of a sensor 21, for example an optical sensor. The type or state of the object can then be determined by means of the control unit 3 and the measured variable x can be used to determine the state of the object. 1 ...x D The input variable can then be determined in such a way that the household appliance is controlled depending on the determined type or determined state of the object. For example, in the case of a washing machine, the washing machine can be controlled depending on the material of the clothes located therein. The input variable u(t) can then be selected depending on the determined material of the clothes.

[0050] In another embodiment, the regulating unit 3 can be used to control a production machine (technical system 2) of a production system by controlling an actuator 22 of the production machine to be controlled via an input variable. The production machine 11 can be, for example, a machine for punching, sawing, drilling, milling, turning and / or cutting.

[0051] The sensor 21 can then be, for example, an optical sensor which detects the properties of the production product. It is possible that the actuator 32 controlling the production machine is actuated according to the determined properties of the production product, so that the production machine correspondingly carries out the subsequent processing steps of the production product. It is also possible that the sensor 31 determines the properties of the production product processed by the production machine and adapts the actuation of the production machine accordingly for the subsequent production product.

[0052] The regulation of the regulation unit 3 follows a regulation strategy. The regulation strategy should be adapted by a dynamic process so that the system behavior becomes optimal in terms of the quality function. For this purpose, an optimization method is implemented, i.e. the model parameters of the regulation model on which the regulation strategy is based are optimized so that the performance of the regulated technical system 2 is optimized. For this purpose, a regulation model (dynamic model) is created in the model creation block 4, which is the basis of the regulation strategy of the regulation unit 3. The model creation block 4 determines the model parameters of the regulation model based on the Bayesian optimization method implemented in the optimization block 5. This is done based on a predetermined quality function determined or predetermined in the quality function block 6.

[0053] In a further preferred embodiment, the adjustment unit 3, the model creation block 4, the optimization block 5 and the quality function block 6 are implemented in a computing unit. The computing unit comprises a control device, one or more processors and at least one machine-readable storage medium on which instructions are stored, which instructions, when executed on the processor, cause the computing unit to implement the invention according to the method.

[0054] The technical system 2 corresponds to a dynamic system which is regulated in an optimized manner by means of a regulation unit 3 with a suitable regulation strategy, which should be created accordingly with the help of a Bayesian optimization method. The Bayesian optimization method is used to determine the regulation model by iteratively applying various test model parameter sets to regulate the technical system 2 during the optimization method and adapting the model parameters based on the resulting state variables. In this case, the quality function is modeled with the help of a Gaussian process regression, by which the regulation performance of the technical system is defined as a function of the model parameters. The performance of the regulation is derived from a predefined quality criterion, which assigns the quality of the regulation to the resulting state variables. The state variables have tolerances, so that the quality function is preferably created by a Gaussian process regression.

[0055] Basically, the problem involves finding a regulation strategy that maps the system state x to the input parameter vector u=π θ (x) Among them in represents the model parameters of the control strategy in the model parameter domain θ. Based on the state vector x and the input variable vector u, a quality function J that depends on the model parameters θ is predetermined over a predetermined time range t=0 ... T, wherein the model parameters θ are to be optimized by an optimization method,

[0056]

[0057] stx t+1 =f(x t , π θ (x t ))+v

[0058] in Corresponding to the expected value, c(x t ,u t ) represents the state vector x t The state given is attached to the input parameter vector u t costs in the case of represents a state transition model which describes the dynamics of the technical system 2 and, among other things, via the noise variables v~N(0, ∑ v ) is imposed.

[0059] The Bayesian optimization method is used to find the optimal model parameters θ of the regulation strategy by minimizing the quality function * The goal is that the control strategy can achieve the best possible control of the technical system 2 using the control unit 3 , wherein the cost minimization determined by the quality function “optimally” relates to the predefined performance of the overall system consisting of the control unit 3 and the technical system 2 .

[0060] The quality function therefore provides the deviation of the behavior of the actual technical system 2 during the time window t=0T from a predefined behavior correspondingly with a cost J. The evaluation of the quality function therefore requires that the technical system 2 be operated in a real environment during the measurement process. Due to the need to actually operate the control system including the technical system 2 and the control unit 3, the evaluation of the cost J becomes very complex, so that the number of measurement processes at the real technical system 2 for evaluating a specific control strategy should be minimized as much as possible.

[0061] The regulation can be created with the aid of an LQR regulator, which requires a linear dynamic model of the system to be regulated. The LQR regulator can be described by a feedback matrix K, and the elements of the feedback matrix can be fully or partially adapted as model parameters for the optimization.

[0062] Let’s assume that the regulation strategy corresponds to π θ Linear state adjustment strategy of (x)=-K(θ)x.

[0063] Linear control strategies have the advantage that they have fewer dimensions than other control models. In addition, linear control strategies make it possible to implement them in a simple manner in the controller and thus increase the efficiency of the Bayesian optimization.

[0064] In conjunction with Bayesian optimization, a linear quadratic regulator, a so-called LQR regulator, can be used, as is known in the art of creating control strategies. In the case of an LQR regulator, the system behavior during the measurement and the interaction with the environment is determined by a controlled operation with a set of varying input variable vectors and the resulting detected state variable vectors. In this case, the system dynamics are determined according to

[0065] f(x t ,u t )≈Ax t +Bu t

[0066] is linearized, and the cost is squared accordingly

[0067]

[0068] By means of these approximations, an LQR feedback matrix can be created in the model creation block 4, which represents the dynamic model and is generally referred to as K=dlqr(A, B, Q, R). The regulation strategy optimization is performed by directly adapting the feedback matrix (K-Learning), wherein a part of the entries or each entry in the feedback matrix corresponds to a model parameter for optimization. Furthermore, only the components of the feedback matrix K corresponding to the system matrices A and B are assumed as model parameters to be optimized, wherein each entry of the matrices A and B corresponds to a model parameter.

[0069] Alternatively, the model parameters to be optimized can be assumed in the feedback matrix K without the matrices Q and R (weighting matrix). Here, it is sufficient to adapt only the diagonal entries of the weight matrix of the following form:

[0070] K QR (θ)=dlqr(A, B, Q(θ), R(θ))

[0071] in and

[0072] Depending on the subordination of the model parameters to the above matrices, these methods are called K-Learning, AB-Learning, and QR-Learning. In the case of Bayesian optimization, the parameter space must be fully covered with respect to the length scale of the quality function in order to find a good estimate of the optimized model parameters.

[0073] However, without prior knowledge, it is difficult to select the value range of the individual model parameters, i.e. the model parameter domain, for optimization. However, this is important for an effective optimization method without an excessive number of measurement processes. Therefore, the above method provides for selecting a suitable search range for the model parameters, so that the number of measurement processes can be reduced. This is necessary in particular in high-dimensional cases, since it is not easy to manually set the value range of each of the model parameters there. Therefore, the value range of the model parameters is first determined by learning a distribution on the dynamic model, and then the distribution is used to select the value range of each of the model parameters. The distribution is obtained by Bayesian linear regression through the recorded data values ​​of the state parameter vector and the input parameter vector in order to obtain an approximate linear model of the system dynamics. This leads to a Gaussian distribution on the model

[0074] p(vec(A,B)|Data)=N(vec(A,B)|μ AB ,∑ AB )

[0075] where μ AB is the maximum a posteriori (MAP) estimate, where the symbol vec(.,.) indicates that the matrices A and B are converted into vectors.

[0076] After selecting the model parameter value range, it is possible that the value range of the model parameters in some dimensions is chosen too conservatively. This can happen, for example, if the scaling parameter B is too small or due to model bias. It can therefore happen that the optimal model parameter vector does not lie within the selected model parameter domain. Therefore, the model parameter domain can be dynamically adapted during the optimization.

[0077] During the process of Bayesian optimization, there is an estimate of the optimal value of the model parameter values, i.e. the minimum of the approximate quality function in the current model parameter domain. If the position of the estimated optimal value resulting from the Bayesian optimization lies at the range boundary of the model parameter domain, then better model parameters are likely to lie outside the current model parameter domain. It is therefore proposed to expand the value range of the model parameters whose values ​​lie at the boundary of the model parameter domain. This dynamic adaptation of the model parameter domain can be implemented in different ways.

[0078] The adaptation of the value ranges of the model parameters makes it possible to dynamically adapt the value ranges during the optimization starting from a limited value range only for those dimensions of the model parameter vector whose optimization satisfies the boundary ranges of the model parameter domain. As a result, the optimization can be performed more efficiently overall, so that the convergence is significantly improved. In addition, potential model errors can be better compensated, so that the optimized system model becomes more powerful. The improved efficiency makes it possible to scale the Bayesian optimization to high-dimensional control strategies.

[0079] In order to create a quality function for the performance of the control system in dependence on the model parameter set, firstly data

[0080] D={θ i ; J(θ i )}, where i = 1…n

[0081] Therefore, an initial Gaussian process model is trained as a quality function, which maps the test model parameters at a certain cost

[0082] μ(θ * )=kK -1 J σ 2 (θ * )=k(θ * ,θ * )-kK -1 k T

[0083] where K corresponds to the covariance matrix, where

[0084] K ij = k(θ i ,θ j ), k=[k(θ 1 ,θ * ),…,k(θ n ,θ * )]und J=[J(θ 1 ), ..., J(θ n )].

[0085] Therefore, the Gaussian process model provides both an expected value, i.e., the cost J, and the uncertainty of that expected value.

[0086] The initial model parameter domain can now be determined from this, since it is difficult to select the value range of the individual model parameters, i.e. the model parameter domain, for the optimization without prior knowledge. The selection of a suitable model parameter domain is important for an efficient optimization method without an excessive number of measurement processes. In step S3, a suitable search range is therefore selected for the model parameters, so that the number of measurement processes can be reduced. This is particularly necessary in high-dimensional cases, since it is not easy to manually adjust the value range of each of the model parameters there. Therefore, the value range of the model parameters is determined based on a previously trained Gaussian process model, and this distribution is subsequently used to select the value range of each of the model parameters.

[0087] This distribution is obtained by Bayesian linear regression from the recorded data values ​​of the state parameter vector and the input parameter vector in order to obtain an approximate linear model of the system dynamics. This leads to a Gaussian distribution on the model

[0088] p(vec(A,B)|Data)=N(vec(A,B)|μ AB ,∑ AB ),

[0089] where μ AB is the maximum a posteriori (MAP) estimate and the notation vec(.,.) indicates that the matrices A and B are converted into vectors.

[0090] Figure 1 The device 1 also comprises an optimization unit. The optimization unit can be arranged in the control device or alternatively separately from the control device. The purpose of the optimization unit is to determine a control model for the regulator 21, by means of which the dynamic technical system 2 can be regulated. To this end, the optimization unit performs an iterative optimization method, by which the control model is created in such a way that a quality function is minimized. The minimization of the quality function can be expressed as:

[0091]

[0092] The initial condition is given by x 0 Pre-given. x t Here it corresponds to the state vector of the system state at time t, and u t corresponds to the input parameter vector at time point t. The cost matrices Q and R are assumed to be positive semidefinite or positive definite. Based on the dynamic behavior f(x t ,u t )=Ax t +Bu t The linear approximation of π is given by assuming a linear state feedback controller u with a regulation strategy π. t =π(x t )=-Kx t In the case of e, use

[0093] K=(R+B T PB) -1 B T PA

[0094] and

[0095] 0=A T (P -1 +BR -1 B) -1 A-P+Q

[0096] An approximate static solution to the above minimization problem is obtained, where the last mentioned equation corresponds to the time discrete algebraic Riccati equation (DARE), which can be solved efficiently by the Kleinman method in terms of P. The linear state feedback controller is described below by the short notation dlqr(A, B, Q, R).

[0097] The solution of the last-mentioned equation is known in control theory as an LQR controller (LQR: Linear Quadratic Regulator). Modeling of an LQR controller often leads to unsatisfactory results due to the linear approximation of the system dynamics, which is usually only a sufficiently accurate approximation near the operating point.

[0098] The search for a regulation strategy is now performed based on the Bayesian optimization method. The Bayesian optimization method provides for Gaussian process regression. Gaussian process regression is a priori unknown function. Nonparametric method for modeling. With the help of Gaussian process regression, both the course of function values ​​and the uncertainty in the prediction of each of the function values ​​can be determined in the case of noisy observations of the system behavior. A Gaussian process can be understood as a distribution over functions and is defined by a prior mean function m(θ) and a covariance function k(θ, θ′). The mean function describes the prior knowledge about the mass function J(θ) to be mapped and is usually assumed to be zero. The covariance function is also called the core and defines the correlation between two function values ​​J(θ) and J(θ′), respectively, where θ, θ′∈Θ.

[0099] Assume n noisy observations: in A prior distribution over the predictions for the measured data can be created in order to obtain a posterior prediction of the quality function at each point θ*∈Θ. The posterior mean and covariance are given by

[0100]

[0101] in

[0102]

[0103] And the symmetric Gram matrix Has item

[0104]

[0105] Gaussian process regression is used to model the behavior of the technical system 2 and to assess its optimality. The behavior of the arrangement consisting of the regulator and the technical system 2 is represented by a quality function, which represents the functional relationship between the model parameters and the costs resulting from the technical system regulated by the regulation strategy defined by the model parameters. The Bayesian optimization method is then applied to optimize the quality function. This quality function is not analytically possible and must therefore be performed repeatedly. Therefore, the effort for optimization should be limited to the smallest possible number of iterations.

[0106] The iteration corresponds to the measurement process of applying the control strategy defined by the model parameters θ· to the actual technical system 2. This results in a new data pair The new data pair is added to the training data of the Gaussian process

[0107]

[0108] After each measurement process, the function is obtained by maximizing To select a new evaluation point, this can be efficiently performed by numerical optimization techniques such as L-BFGS. Instead of obtaining a function, for example, probability of improvement (P1), expected improvement (E1) and upper confidence bound (UCB) can be used. All these functions provide a trade-off between adopting, i.e., preferring, a range of input parameter vectors in which the quality function has not been evaluated, and adopting, i.e., preferring, a range in which the estimated optimal value (minimum value) of the quality function lies.

[0109] For Bayesian optimization, only the overall behavior of the technical system running through the measurement process is evaluated, while the trajectory data or process data are usually discarded. For system identification with the help of Bayesian linear regression, the state parameters can be sampled at a high frequency to obtain uncertainty measures for the model to be estimated.

[0110] In the classical parametric regression case, a model y(x, w) is assumed with inputs x and weights or parameters w, which are used to estimate a noisy target variable t.

[0111] Assuming the noise follows a Gaussian distribution with precision (inverse variance) γ, the posterior distribution corresponds to the matrix The n measurements described in the vector t n =[t1 , ..., t n ] T The weights of the corresponding target values ​​are described in:

[0112]

[0113] where n corresponds to the number of data points, γ corresponds to a predetermined constant describing the noise in the data, and m 0 Corresponding to the a priori mean of the model parameters to be estimated (the mean of the Gaussian prior of the model parameters), S 0 corresponds to the prior covariance of the model parameters to be estimated (the covariance of the Gaussian prior of the model parameters), X corresponds to the location of the data points (summarized in a matrix), t corresponds to the function value of the data points (summarized in a vector), and m n corresponds to the posterior expected value (mean value) of the model parameter to be estimated and S n corresponds to the posterior covariance of the model parameters to be estimated.

[0114] The maximum a posteriori estimate (MAP estimate) of the weight w corresponds to the mean of the posterior distribution, that is, w MAP =m n and its covariance corresponds to

[0115] The control model is determined by reinforcement learning in order to determine the parameterization of the linear state feedback controller by means of a Bayesian optimization method.

[0116] Measures for accelerating the convergence of the optimization process are proposed below.

[0117] As a preliminary assumption, it is assumed that the nonlinear technical system 2 is approximated by a linear model and that the Bayesian optimization method is used to optimize the terms of the system matrices A, B. The resulting regulation model π AB Then it can be written as:

[0118] π AB (x t ; θ) = -K AB (θ)x t , K AB (θ)=P=dlqr(A(θ), B(θ), Q, R)

[0119] This method is called AB learning method.

[0120] An alternative learning method is the so-called K-learning method, in which the optimization is directly performed into the feedback gain matrix. Here, the regulation model corresponds to π K (x t ; θ) = -K K (θ)x t, which, in contrast to the previously described approach, does not use the above method and is therefore a model-free approach.

[0121] according to Figure 2 The method for creating the regulation model is now described with the aid of a Bayesian optimization method.

[0122] In step S1, the MAP weight matrix A is first determined MAP , B MAP This can be done without prior knowledge about the regulation model. In the identification process of a technical system, the goal is to determine the current state x given by t and the input parameter vector u t Determine the subsequent state x of technical system 2 t+1 . Therefore the following applies:

[0123]

[0124] The maximum a posteriori estimate (MAP estimate) corresponds to the weight θ MAP .

[0125] The advantage of using Bayesian linear regression is that it not only receives the mean value θ MAP MAP estimate, and also receives uncertainty σ MAP These correspond to the matrix S n The diagonal items of .

[0126] In step S2, the value ranges of the model parameters to be determined of the control model are determined. The Bayesian optimization must also cover the value range of the model parameters as fully as possible with respect to the length scale of the quality function. A priori, it is usually difficult to determine in which value ranges the model parameters, i.e. the entries of the matrix K, must be optimized in order to achieve a good control behavior.

[0127] To this end, the MAP estimate w from the system identification in step S1 is used. MAP , σ MAP , a meaningful value range is selected for each of the model parameters, within which the model parameter is subsequently optimized.

[0128] For the AB learning process, the MAP estimate θ from the matrix K MAP , σ MAP Starting from the variance of the parameter estimates, the choice of the range corresponds to Θ (the model parameter domain). Therefore, the iσ variance around the MAP estimate can be assumed to be the range of the parameter values, so the following applies:

[0129]

[0130] Here, i can preferably be assumed to be between 1 and 4, in particular 2.

[0131] Thus, during the Bayesian optimization method, model parameters with higher uncertainty are assigned a larger value range for optimization and, conversely, well-identified model parameters, ie model parameters with low uncertainty, are assigned a smaller value range.

[0132] In the case of the K-learning method, a value range is constructed for the estimation system with the aid of an LQR controller:

[0133] K MAP =dlqr(A MAP , B MAP , Q, R).

[0134] The value range of the model parameters of the regulation model can in this case be set between 0 and twice the MAP estimate value of the corresponding model parameter

[0135]

[0136] In general, value ranges for the model parameters of the control model can be predefined:

[0137]

[0138] in

[0139] In step S3, an a priori mean function is determined. This uses an approximate model of the technical system 2 to determine a simple simulator of the technical system 2. The simulator is used to estimate the form of the quality function. The approximate model of the technical system 2 is determined, for example, by means of a Gaussian process based on several measurement points. The number of measurement points used for this purpose can be between 10 and 100. An approximate a priori mean function of the cost can then be constructed, for example, by Gaussian process regression.

[0140] Then, in step S4, the initial model parameters θ are determined by minimizing the prior mean function 0 .

[0141] With the help of the initial model parameters θ 0 The control model thus defined can be evaluated in step S5 during the measurement process and the corresponding minimum cost can be determined based on the quality function

[0142] Then, in step S6, taking into account the last determined data pair In the case of obtaining function The next model parameter θ for the next iteration of the optimization method is determined by maximizing n+1 .

[0143] With the help of the current model parameters θ n+1 , the control model thus defined can be evaluated in step S7 for the next measurement process and the corresponding minimum cost can be determined

[0144] Then, in step S8, the final determined data pair is added to the training data

[0145]

[0146] Then, in step S9, a termination criterion is checked, which for example indicates whether sufficient performance of the control model has been achieved or whether sufficient convergence exists. If the termination criterion is met (either: yes), the method ends with step S10, otherwise (either: no) the system jumps back to step S4.

[0147] In step S10 , the finally determined model parameters are applied to the regulation strategy.

Claims

1. A method for determining a control strategy for a technical system (2) by means of a Bayesian optimization method, wherein the control strategy is created and can be implemented based on model parameters θ of a control model, wherein the following steps are performed for optimal control: - providing a quality function, the form of which corresponds to a regression function and which evaluates the regulation quality of the technical system (2) based on the model parameters θ; - performing a Bayesian optimization method based on the quality function in order to iteratively determine a model parameter set having the model parameter θ within a model parameter domain θ describing a range of permissible values ​​of the model parameter θ; and -According to the maximum a posteriori estimate of the quality function θ MAP To determine a model parameter domain Θ of at least one of the model parameters.

2. The method according to claim 1, wherein the parameter regression model maps the system state and the input parameter vector of the technical system (2) to the subsequent system state and is trained accordingly to obtain the weighting matrix w MAP .

3. The method according to claim 1 or 2, wherein the model parameters θ are determined by means of an AB learning method for an LQR controller, wherein for at least one maximum a posteriori estimate θ MAP The uncertainty measure is determined separately, where the value range of at least one model parameter θ is specified to be within the maximum a posteriori estimate θ MAP around, In the AB learning method, it is assumed as a preliminary assumption that the nonlinear technical system is approximated by a linear model, and the Bayesian optimization method is used to optimize the terms of the system matrices A and B, where the resulting regulation model π AB Then it can be written as: π AB (x t Λθ)=-K AB (θ)x t ,K AB (θ)=P=dlqr(A(θ),B(θ),Q,R)。 4. The method according to claim 3, wherein the uncertainty σ of the predetermined expected value is MAP In the case where at least one model parameter θ has a value range determined within the maximum a posteriori estimate θ MAP around.

5. The method according to claim 1 or 2, wherein the model parameters θ are determined by means of a K-learning method for an LQR regulator, wherein the value range of at least one model parameter θ is defined within a maximum a posteriori estimate θ MAP around, which is directly optimized into a feedback gain matrix in the K-learning method, where the regulation model corresponds to π K (x t ; θ) = -K K (θ)x t , which is a model-free solution.

6. The method according to claim 5, wherein the maximum a posteriori estimate θ MAP The value range of at least one model parameter θ is determined by the following metric, which is determined as a predetermined factor between 0 and 1 and the associated maximum a posteriori estimate θ MAP The product of .

7. The method according to any one of claims 1 to 2, wherein the optimization method starts with initial model parameters obtained by minimizing a priori mean value function, wherein the non-parametric approximation model of the technical system (2) is trained to obtain the priori mean value function.

8. A device for determining a control strategy of a technical system (2) by means of a Bayesian optimization method, wherein the control strategy is created and can be implemented based on model parameters θ of a control model, wherein the device is designed to carry out the following steps for optimal control: - providing a quality function, the form of which corresponds to a regression function and which evaluates the regulation quality of the technical system (2) based on the model parameters θ; - performing a Bayesian optimization method based on the quality function in order to iteratively determine a model parameter set having the model parameter θ within a model parameter domain θ describing a range of permissible values ​​of the model parameter θ; -According to the maximum a posteriori estimate of the quality function θ MAP To determine a model parameter domain Θ of at least one of the model parameters.

9. A control system (1) having a technical system (2) and a control unit (3) for controlling the technical system (2), wherein a control model for providing an input variable vector as a function of state variables of the technical system (2) is implemented in the control unit (3), wherein a model creation block (4) is provided for determining model parameters θ of the control model based on a Bayesian optimization method implemented in an optimization block (5), wherein a control strategy is created and can be implemented based on the model parameters θ of the control model, wherein the following steps are implemented for optimizing the control: - providing a quality function, the form of which corresponds to a regression function and which evaluates the regulation quality of the technical system (2) based on the model parameters θ; - performing a Bayesian optimization method based on the quality function in order to iteratively determine a model parameter set having the model parameter θ within a model parameter domain θ describing a range of permissible values ​​of the model parameter θ; - determining a model parameter field Θ for at least one of the model parameters θ as a function of the associated maximum a posteriori estimate θMAP of the quality function. 10 . A computer program product comprising a computer program which is configured to carry out the method according to claim 1 , when the computer program is executed on a computing unit.

11. A machine-readable storage medium having a computer program stored thereon, the computer program being configured to implement the method according to any one of claims 1 to 7 when the computer program is executed on a computing unit.