Method for controlling or regulating a technical system

Abstract

A method for controlling or regulating a technical system is described, wherein a control or regulation parameter is determined by Bayesian optimization and the next test point is determined in the Bayesian optimization by means of an acquisition function. The acquisition function in the Bayesian optimization is automatically determined by a learning method, the control or regulation parameter is determined in the Bayesian optimization by means of the determined acquisition function, and the technical system is controlled or regulated as a function of the determined parameters.

Description

Technical Field

[0001] The invention relates to a method for controlling or regulating a technical system and to a computer program provided therefor. Background Art

[0002] Bayesian Optimization can be used as a global optimization method for black box processes, in particular for optimizing the parameters of a function corresponding to an underlying technical process, wherein the technical process is defined by its inputs and outputs. To this end, test points can be selected iteratively in Bayesian Optimization, the goal of which is often to find the input parameters that achieve the optimal value of the function with as few tests as possible.

[0003] For this purpose, Bayesian optimization uses so-called acquisition functions to select the next test point. These acquisition functions are typically defined manually, see E. Brochu, VM Cora, and Nando deFreitas, "A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning", CoRR 2010. Summary of the invention

[0004] A method for controlling or regulating a technical system, in particular a tool, a machine tool, a robot or a vehicle, is proposed. In this case, at least one parameter of the control or regulation is determined by Bayesian optimization, wherein in the Bayesian optimization the next test point of the iterative optimization process is determined by an acquisition function. It is now proposed that the acquisition function in the Bayesian optimization can be automatically determined by a learning process, preferably by reinforcement learning or imitation learning, wherein the parameters of the control or regulation are then determined in the Bayesian optimization with the aid of the determined acquisition function and the technical system is controlled or regulated as a function of the determined parameters.

[0005] In this case, an acquisition function is, in particular, a comparative evaluation function for selecting (potential or actual) future test points.

[0006] Bayesian Optimization is a gradient-free method for global optimization of an unknown function by iterative evaluation of test points.

[0007] Reinforcement learning is a machine learning framework for optimizing sequential decision problems. In this case, a so-called strategy is a decision function that is to be optimized by reinforcement learning.

[0008] The acquisition function is preferably learned in a simulation of the control or regulation of the technical system, wherein in the simulation an optimization for the control or regulation is selected by Bayesian optimization as a training task for the learning method.

[0009] In the simulation, conditions or environmental parameters for controlling or regulating the technical system are preferably randomly selected. Using an initial acquisition function, a process quality for optimizing the control or regulation of the technical system using Bayesian optimization is preferably determined in the simulation, and the initial acquisition function is adapted or optimized as a function of the determined process quality, and an improved acquisition function is thus learned. When using the learned acquisition function, the process optimization is preferably carried out in the simulation on the actual technical system after the acquisition function has been determined.

[0010] The learning method for acquiring the function is preferably carried out iteratively until a termination criterion is reached, in particular a predetermined process quality, a predetermined duration of the method or a predetermined number of iterations.

[0011] The present invention enables automatic, self-learning creation of acquisition functions. These learned acquisition functions are generally more powerful and can be evaluated faster. In addition, these acquisition functions can be learned in an optimized manner for specific problem categories or requirements, thereby enabling additional efficiency advantages, such as a smaller number of test points. The acquisition functions enable more efficient and more effective searches, that is, the optimal value is found with fewer attempts, and the optimal value found has a smaller degree of regret (regret, the difference from the optimal function value). The computing time generally required for the evaluation of the learned acquisition functions is significantly less than some of the more complex commonly used acquisition functions such as "entropy search" (compared to several hours). <ls)。 BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention will be explained in detail below with reference to the accompanying drawings and by means of embodiments. Figure 1 An exemplary sequence of a method for controlling or regulating a technical system is schematically shown. DETAILED DESCRIPTION

[0013] Bayesian Optimization can be used as a global optimization method for black-box procedures. To this end, test points can be selected iteratively, with the goal usually to find the input parameters that achieve the optimal value of the function with as few tests as possible. For this purpose, Bayesian Optimization uses so-called acquisition functions to select the next test point.

[0014] Known acquisition functions include:

[0015] - “Probability of Improvement”: In practice, this acquisition function may lead to ineffective local search;

[0016] - "Expected Improvement": This acquisition function tries to compensate for the shortcomings of the "likelihood of improvement" function, but as a result the search is very wide and may not find the optimum, but may jump back and forth between parameter ranges;

[0017] - "Upper Confidence Bound": This acquisition function can be very effective, but often provides relatively poor results for high-dimensional problems;

[0018] - "Entropy Search": This acquisition function provides very good results in many cases, but often requires very long computation times (often hours to days).

[0019] It should generally be maintained that general advantages and disadvantages are found for known acquisition functions, but these advantages and disadvantages can only be used as a basis for selecting an optimization for a specific technical process, and not as a definitive decision point. That is, in practice, a plurality of acquisition functions are typically tested in order to select the corresponding best acquisition function for the problem.

[0020] A method is now proposed with which the acquisition function to be used can be automatically determined by means of a learning method. The acquisition function can be learned in this case, in particular by providing an optimization of an unknown function by Bayesian optimization as a training task for the learning method.

[0021] For this purpose, in a first step, a representation of the relevant function class of the technical system or process is preferably known or provided. This can be done by predefining a general function class, such as, for example, a polynomial, or by a special simulation of the technical system to be controlled or regulated under different conditions.

[0022] If a corresponding method is used for controlling or regulating a technical system, for example for optimizing a grinding process, system properties such as the input material, the specific grinding method or the environmental conditions of the grinding process can be represented in a corresponding simulation.

[0023] In a second step, the optimal acquisition function for the process to be optimized is learned in a simulated manner. To this end, the following steps are preferably performed repeatedly:

[0024] a. First, conditions such as environmental conditions can be randomly selected for use in a fixed simulation environment;

[0025] b. Using Bayesian optimization, find the optimal value of the function under the current strategy (collection function);

[0026] c. Ability to learn improved strategies based on the ascertained data.

[0027] The strategy learned in the second step can now be used as a learning function for the actual system in order to optimize the technical process.

[0028] In the learning method of the second step, during the simulation, the step indicated by i of the Bayesian optimization is preferably performed for each iteration k of the learning method:

[0029] -GP_k,i: the process model learned at time i,

[0030] -\theta_k,i: the selected test point,

[0031] -q_k,i: The resulting process quality.

[0032] With the help of this data, policies can be learned either through reinforcement learning or imitation learning.

[0033] Free learning (reinforcement learning):

[0034] In the free learning mode, no target acquisition function is predetermined; in other words, a completely new acquisition function is to be learned. In this case, so-called model-free reinforcement learning methods are used to train a policy that represents the optimal acquisition function. Such methods are, for example, policy gradient methods, Q-learning methods, actor-critic methods. For each of these methods, a large number of algorithms are available that come into consideration for the described learning process.

[0035] The advantage of free learning is that a generally better solution ultimately results, which can also be tailored to the desired application category in order to achieve efficiency advantages there.

[0036] Goal-directed learning (imitation learning):

[0037] In the case of goal-oriented learning, an existing acquisition function is predetermined as a goal. The advantage is that the learned representation of the acquisition function can be evaluated much more quickly. For this purpose, data are generated with the aid of the goal acquisition function, which are then used in a supervised learning method to train the strategy to reproduce the observed behavior.

[0038] The advantage is a much simpler learning problem, since only known behaviors need to be imitated and no new behaviors need to be learned at all. This may also reduce the learning time overall.

[0039] If reinforcement learning is used, the policy can be adapted by reinforcement methods after each iteration k. If imitation learning is used, the policy is adapted only after all data points have been established.

[0040] The proposed method can be used, for example, to optimize various technical control or regulation processes, in particular to control or regulate technical systems, in particular robots, vehicles, tools or machine tools, in order to determine and use output variables for control or regulation based on the collected input variables. As input variables for the regulation or control, sensor data or variables known based on sensor data are particularly considered. The sensor data can originate from a sensor of the technical system or can be received from the outside by the technical system. Based on the output variable or the control or regulation variable, at least one actuator of the technical system is controlled or regulated with a control signal by a computing unit. For example, the movement of a robot or vehicle can be controlled in this way or the drive unit or driver assistance system of the vehicle can be controlled. More specific uses include, for example, the optimization of grinding processes, the optimization of ultrasonic welding processes, the optimization of joining processes, the optimization of drilling processes (e.g., deep hole drilling processes), the adjustment of PID controllers, the optimization of parameterized desired trajectories of vehicles or robots, and the like.

[0041] For example, given a (not accurate enough) simulation of a given grinding process, the actual grinding process is optimized. The simulation can be varied in its behavior by means of different control variables. The control variables can be, for example, tool material, workpiece material, environmental variables, process noise, etc. With the aid of the described method, an optimal or optimized acquisition function for a general grinding process can be learned from a given simulation by regularly varying the simulation parameters during the learning process. Such an acquisition function leads to a particularly efficient Bayesian optimization process for parameterizing the actual grinding process, since the underlying acquisition function is specific to the grinding process (or a certain class of grinding processes).

[0042] Figure 1An exemplary sequence of the proposed method for learning an acquisition function for Bayesian optimization is schematically shown.

[0043] In a first step 1, the strategy (acquisition function) is initialized. In the following step 2, environmental conditions, such as material parameters, are determined or randomly selected for simulation. For these environmental conditions, the simulation of the technical process, here the grinding process, is started in the following step 3. In this case, the first counter is first set to zero in step 4. In the following step 5, the process model is initialized according to the counter value zero of the first counter. In the following step 6, the process model is provided according to the current counter value of the first counter. The current strategy selects process parameters according to the provided process model in step 7. In the following step 8, the process parameters are identified in the simulation or evaluated according to the simulated process quality. Accordingly, in the following step 9, the process model is updated or adapted according to the identification or evaluation situation. In the following step 10, the first counter is increased by one step. In the following step 11, the loop of steps 6 to 10 is ended according to the termination criteria (such as process quality or the number of iterations), and a branch is made in step 12 or a branch is made to step 6, and steps 6 to 10 are continuously executed again. In step 12, a reward function is evaluated, which assigns a reward value (Reward) as a function of the determined process quality for the selected parameters. In a subsequent step 13, the policy (acquisition function) is updated by a learning method (e.g., reinforcement learning). Subsequently, in a subsequent step 14, it is determined whether the simulation is to be performed again in another loop by branching in step 2, or whether the simulation is to be terminated (e.g., with sufficient convergence of the learning method) and branching is performed in step 15.

[0044] After the previous simulation, the process optimization on the actual system is started with step 15. In step 15, the second counter is initialized or set to zero. In the following step 16, the process model corresponding to the counter level zero of the second counter is initialized. In the following step 17, the process model is provided according to the current counter value of the second counter. In step 18, the current strategy corresponding to the current counter value of the second counter selects the process parameters according to the provided process model. In the following step 19, the process parameters are identified in the actual system or evaluated according to the process quality on the actual system. Accordingly, in the following step 20, the process model is updated or adapted according to the identification or evaluation. In the following step 21, the second counter is increased by one step. In the following step 22, the loop of steps 17 to 22 is terminated according to a termination criterion (for example, process quality or number of iterations), and in step 23 a branch is made or a branch is made to step 17, and steps 17 to 22 are executed again in succession.

[0045] In step 23 , optimal or optimized parameters are extracted for the optimization process and the technical process, here a grinding process, can be carried out with the optimal parameters, ie the technical system is controlled or regulated using the determined parameters.

[0046] The described method can also be used to optimize a machine learning system, in particular a neural network. In this case, the parameters of the learning system, in particular hyperparameters (Hyperparameters), can be determined by Bayesian optimization. In Bayesian optimization, the next test point is determined by an acquisition function, the acquisition function in the Bayesian optimization is automatically determined by a learning method, and the parameters are learned in the Bayesian optimization with the aid of the determined acquisition function. The machine learning system (das maschinelle Lernsystem) is then trained according to the learned parameters. The machine learning system, such as a neural network, is implemented in this case in particular by a computer, that is, not only the calculations for determining the acquisition function but also the calculations for the training process are performed by a processor. The parameters of the learning system are stored in a machine-readable memory.

Claims

1. A method for controlling or regulating a technical system, wherein parameters of the control or regulation are determined by Bayesian optimization, wherein in the Bayesian optimization the next test point is determined by an acquisition function, characterized in that Automatically determining an acquisition function in a Bayesian optimization by means of a learning method, wherein the acquisition function is determined by reinforcement learning or imitation learning, parameters of a control or regulation are determined in the Bayesian optimization by means of the determined acquisition function, and the technical system is controlled or regulated as a function of the determined parameters, wherein no target-acquisition function is predetermined in the reinforcement learning, wherein a model-free reinforcement learning method is used to train a policy representing an optimal acquisition function; In imitation learning, an existing acquisition function is predetermined as a target, data being generated with the aid of the target acquisition function and then used in a supervised learning method to train the policy to reproduce the observed behavior.

2. The method according to claim 1, characterized in that: The acquisition function is learned in a simulation of a control or regulation of the technical system.

3. The method according to claim 2, characterized in that In the simulation, optimization for the open-loop or closed-loop control is selected by Bayesian optimization as the training task for the learning method.

4. The method according to claim 2 or 3, characterized in that: In the simulation, the conditions for controlling or regulating the technical system are selected randomly.

5. The method according to claim 2 or 3, characterized in that: In the simulation, an initial acquisition function is used to determine a process quality for optimizing a control or regulation of the technical system using Bayesian optimization, and the initial acquisition function is adapted as a function of the determined process quality.

6. The method according to claim 2 or 3, characterized in that: In the simulation, after the acquisition function has been determined, a process optimization is carried out on the actual technical system.

7. The method according to any one of the preceding claims 1 to 3, characterized in that The learning method is carried out iteratively until a termination criterion is reached.

8. The method according to any one of the preceding claims 1 to 3, characterized in that The technical system is a tool, a machine tool, a robot or a vehicle.

9. The method according to claim 7, characterized in that The termination criterion is the achievement of a predetermined process quality, a predetermined duration of the method or a predetermined number of iterations.

10. A method for optimizing a machine learning system, wherein the parameters of the learning system are determined by Bayesian optimization, wherein the next test point is determined by an acquisition function in the Bayesian optimization, characterized in that Automatically determining an acquisition function in Bayesian optimization by a learning method, wherein the acquisition function is determined by reinforcement learning or imitation learning, parameters are learned in Bayesian optimization by means of the determined acquisition function, and the machine learning system is trained according to the learned parameters, wherein no target-acquisition function is predetermined in the reinforcement learning, wherein a model-free reinforcement learning method is used to train a policy representing an optimal acquisition function; In imitation learning, an existing acquisition function is predetermined as a target, data being generated with the aid of the target acquisition function and then used in a supervised learning method to train the policy to reproduce the observed behavior.

11. The method according to claim 10, characterized in that The method is used to optimize a neural network.

12. A computer program product arranged to carry out the method according to any one of the preceding claims.

13. A computing unit having a computer program product according to claim 12.

Images