Method for optimizing a policy of a robot

Abstract

Method for optimizing a policy of a robot. A method for optimizing a predefined policy of a robot, wherein the policy is a Gaussian mixture model. The method starts with initializing a Gaussian process, wherein the Gaussian process has at least one kernel k that obtains as input arguments a distance determined between probability distributions characterized by a Gaussian mixture model and a Gaussian process, respectively, and according to a probabilistic product kernel. This is followed by optimizing the Gaussian process such that the Gaussian process predicts a cost according to parameters of the Gaussian mixture model. This is followed by determining optimal parameters of the Gaussian mixture model according to the Gaussian process, wherein parameters are selected according to the Gaussian process such that the Gaussian process outputs an optimal cost function.

Description

TECHNICAL FIELD

[0001] The invention relates to a method for optimizing a policy of a robot implemented by means of a Gaussian Mixture Model, and to a device, a computer program and a machine-readable storage medium. BACKGROUND

[0002] Gaussian Processes (GP) and their optimization by means of Bayesian Optimization (BO) are known from the authors Shahriari, Bobak et al. "Taking the human out of the loop: A review of Bayesian optimization." Proceedings of the IEEE 104.1 (2015): 148-175.

[0003] Gaussian Mixture Models (GMM) for policy adjustment of robots are known from the authors Calinon, Sylvain. "A tutorial on task-parameterized movement learning and retrieval." Intelligent service robotics 9.1 (2016): 1-29.

[0004] Probability Product Kernels (PPK) are known from the authors Jebara, Tony et al. "Probability product kernels." Journal of Machine Learning Research 5 July (2004): 819-844.

[0005] ADVANTAGES OF THE INVENTION

[0006] Bayesian Optimization (BO) can efficiently optimize complex robot problems. However, a disadvantage of Bayesian Optimization is that it usually does not scale well to high-dimensional problems, e.g. direct optimization of parameters of a policy for a robot implemented by means of a Gaussian Mixture Model. Therefore, only a limited small number of mixture components of the Gaussian Mixture Model can be optimized. The inventors propose to mitigate the so-called curse of dimensionality by making full use of the structure of the policy and for this propose special kernels.

[0007] The kernel according to the present invention has the advantage that on the one hand the search space is significantly reduced by making full use of the symmetry of the search space and on the other hand also the explainability of the policy is increased.

[0008] Because the kernel according to the present invention effectively reduces the dimensionality, it is thus possible to efficiently optimize a high-dimensional Gaussian Mixture Model policy, e.g. by BO. Since now a high-dimensional Gaussian Mixture Model policy can be optimized, it is finally also possible to provide a robot with a more precise and more reliable policy. SUMMARY

[0009] In a first aspect, the present invention relates to a computer-implemented method for optimizing a given policy of a robot, wherein the policy is a Gaussian Mixture Model (GMM) outputting at least one sub-trajectory depending on a start / and target position / state of the robot. The given policy is preferably sub-optimal, i.e. the policy is already usable, but is not optimal in terms of a given criterion, such as a cost. It can thus be said that the policy is already partially set up for outputting sub-trajectories. A sub-trajectory can be understood as the sub-trajectory being a segment of a complete trajectory, wherein the complete trajectory characterizes a path or motion of the robot such that the robot reaches a target position using the complete trajectory. The sub-trajectory can be a segment of a path or a (sub)motion, in particular an action of the robot, wherein a plurality of sequentially combined sub-trajectories result in a complete trajectory. The sub-trajectory preferably corresponds to an action of the robot.

[0010] The method starts with initializing a Gaussian Process, which is suitable for estimating a cost depending on a parameterization of the GMM, in particular for determining a cost the robot has to spend in order to reach a target position. The parameterization of the GMM is understood as the parameters of the GMM. The parameters of the GMM are the expectation values and variances of the individual mixture components of the GMM. The cost can characterize a robot expenditure the robot has to make in order to reach a target position or state. Additionally or alternatively, the cost can characterize a penalty that would occur to the robot if the robot chooses a bad sub-trajectory. The cost is preferably calculated using a cost function.

[0011] The Gaussian Process has at least one kernel k, which describes a similarity between two input arguments. The kernel is also known in the terminology of the covariance function of the GP. In many cases, this similarity is a function of the distance of the two input arguments. In the case of the present invention, the input arguments are described by the GMM. For this purpose, the distance between two GMMs is determined based on the PPK in terms of a metric.

[0012] Then a plurality of trajectories is created according to the policy, preferably one trajectory is determined separately for each policy. Here, sub-trajectories can be determined successively and then combined into a trajectory. It is conceivable that this action is repeated a plurality of times in order to thereby determine a plurality of trajectories. However, it is also conceivable that a plurality of trajectories is created according to GMMs which are parameterized in a slightly changed manner, wherein for this the parameters of the GMMs are changed slightly, in particular randomly spread around the given parameter values of the respective mixing components. It is conceivable that the parameters are changed for a pre-given mixing component or for all mixing components. Then a cost is determined for the plurality of trajectories. It can be said that a plurality of different policies, i.e. different parameterizations of the GMM, are rolled out and for each rollout a cost is determined.

[0013] Then the GP is optimized such that it substantially predicts the cost to which the respective trajectory belongs according to the used parameters of the GMM for the plurality of trajectories. For this preferably a BO is used.

[0014] Then a new parameter for the GMM can be determined by optimizing an Akuisitionsfunktion. Then the derived policy can be rolled out with the new parameter of the GMM and its cost can be determined. Thus, at least one further trajectory is created and its cost is determined. Then the GP can be adapted, in particular retrained, with the new data. The steps of determining a new parameter and retraining the GP can be repeated a plurality of times until, for example, the GP has reached a sufficient prediction quality or a maximum number of experiments has been reached.

[0015] Then an optimal parameter for the GMM is determined according to the GP, such that the GP outputs an optimal cost for the optimal parameter, and then the parameters of the policy are replaced by the optimal parameter.

[0016] It is proposed to determine the inner product between a data point q of the GMM p and the GP p by means of The distance is calculated according to the PPK. It should be noted that the data point (in the following also referred to as support point of the GP) represents the previously considered GMM for creating the trajectory. In simple terms, this inner product is referred to in the following as the inner product between the GMM and the GP. Preferably, the inner product is the L 2 inner product.

[0017] Furthermore, it is proposed to determine the inner product as a sum over a plurality of normal distributions. The plurality of normal distributions comprises a normal distribution for each possible combination between a mixture component of the GMM and a support point of the GP, respectively. The plurality of normal distributions is parameterized such that the normal distributions have as parameters, in case of a respective expected value of a mixture component giving an expected value of a support point, and as variances, a sum over the variances of the mixture component and the support point under consideration, respectively. The plurality of normal distributions is weighted with the weights of the mixture component and the support point of the GP, respectively. A weight (English: mixture weight) can be assigned to each mixture component of the GMM and to each support point of the GP.

[0018] Furthermore, it is proposed that the distance between the input arguments p and q of the kernel k is an inner product composed of the difference between the GMM and the GP squared.

[0019] Furthermore, it is proposed to solve the inner product according to a second binomial formula and to determine the individual terms of the solved second binomial formula as a sum over a plurality of normal distributions, as described above. The second binomial formula is: wherein for identical terms , then a combination composed of a mixture component or a support point is used, respectively.

[0020] Furthermore, it is proposed to assign a length scale (English: lengthscale) to a mixture component of the GMM and to a support point of the GP, respectively, wherein the distance is scaled, in particular normalized, according to the length scale.

[0021] Furthermore, it is proposed to recreate trajectories after optimizing the GP. For this purpose, parameters for the GMM are determined from the GP. An acquisition function uses the GP to determine the parameters of the GMM. The GMM is then parameterized with the parameters so that further trajectories are then created by means of the newly parameterized GMM. The costs are also calculated for the further trajectories. The GP is then re-optimized according to the further trajectories and the associated costs.

[0022] Dynamic domain adaptation (English: D ynamic D omain Adaption, DDA) is used for optimization. For more details on this, see the authors Fröhlich, Lukas P., et al. "Bayesian Optimization for Policy Search in High-Dimensional Systems via Automatic Domain Selection." arXiv preprint arXiv:2001.07394 (2020), which is available online at: https: / / arxiv.org / abs / 2001.07394.pdf. This has the advantage that a particularly efficient method is implemented by using the DDA.

[0023] It is furthermore proposed that the trajectory of the robot is determined in accordance with the policy with replaced parameters, in particular in accordance with the sub-trajectory obtained therefrom, to manipulate the robot. Preferably, the robot is provided with control variables in accordance with the policy.

[0024] In other aspects, the present application relates to a device and a computer program, respectively, which are set up to carry out the above-described method, and to a machine-readable storage medium on which a computer program is stored. BRIEF DESCRIPTION OF DRAWINGS

[0025] Embodiments of the present application are explained in more detail below with reference to the enclosed drawings. In the drawings:

[0026] Figure 1 A flow chart schematically illustrating an embodiment of the present application;

[0027] Figure 2 A flow chart schematically illustrating an embodiment for controlling an at least partially autonomous robot;

[0028] Figure 3 A flow chart schematically illustrating an embodiment for controlling a production system. DETAILED DESCRIPTION

[0029] The behavior of complex autonomous systems, such as manipulating robots, is typically coded by so-called policies (English: policy) which map the current robot state s to a desired action a. Different parameterizations for policies have been proposed: neural networks, dynamic motion primitives and Gaussian mixture models (GMM). Especially for manipulating robots, GMM-based policies have proven useful due to their flexibility for capturing complex motions and a relatively small number of parameters.

[0030] A typical way of "training" the policy is kinesthetic demonstration, i.e. an expert physically grabs the robot and demonstrates the desired behavior (a process called "learning from demonstration"). Based on the recorded behavior, especially the trajectory, the behavior can then be encoded by means of a GMM. After training the policy, the resulting behavior of the robot can still be suboptimal and has to be further coordinated in order to fully achieve the desired goal. Bayesian optimization (BO) can be used for this subsequent coordination step. The suboptimal behavior of the policy can also arise due to the policy having to be adapted for a new task or other conditions prevailing, such as the initial position being changed.

[0031] If an unknown function should be optimized, BO is typically used, where there is no analytical form for the unknown function, but the unknown function can only be analyzed selectively. Since the analysis requires a lot of time and / or resources, each analysis of the function is expensive.

[0032] Due to the high cost of each evaluation, it is desirable to proceed in a random sampling manner and thus also as efficiently as possible in order to find the optimal value of the function. One of the main components of BO is a so-called surrogate model, which approximates the function based on previous observations. For the surrogate model, a Gaussian process (GP) is preferably used. The Gaussian process can be optimized by means of BO, such that the Gaussian process is based on earlier evaluation points and the corresponding observations learns their association and for a subsequently given evaluation point a prediction can be made with respect to the observation .

[0033] A key property of the GP is that it captures the uncertainty in the prediction, i.e. near the data, the prediction uncertainty is small, while the prediction uncertainty increases away from the data.

[0034] The surrogate model can be used to find the next optimal evaluation point by optimizing a so-called acquisition function. The acquisition function trades off exploration (i.e. points with high prediction uncertainty are preferred) and exploitation (i.e. points with good prediction results are preferred). In recent years, many different acquisition functions have been proposed. If the optimal value of the acquisition function has been found, the objective function is evaluated here and the new data ; updates the GP. The method is repeated until either a good enough solution has been found or the maximum number of optimization steps has been reached.

[0035] Disadvantageously, BO cannot be directly scaled to high-dimensional search spaces.

[0036] To this end, the inventors propose to realize the scaling of the BO onto the higher dimensional parameter space by designing a specialized kernel function k that realizes the additional domain knowledge.

[0037] For many scenarios, the kernel function f depends only on the distance d between two elements, i.e. .

[0038] Typically, the Euclidean distance in the parameter space does not work well to correctly detect differences in behavior between two strategies. Since the Euclidean distance is not invariant to differently parameterized strategies.

[0039] Based on this, the inventors propose a kernel function k set for GMMs for use in GPs. The kernel function is based on a Probability Product Kernel (PPK) that defines an inner product between two probability distributions p and q:

[0040] Equation (1)

[0041] It should be noted that p and q are GPs and GMMs.

[0042] Further, it is proposed to manipulate the PPK kernel such that it is a stationary kernel. To be able to use the properties of known kernels, such as the Matern52 kernel, it is proposed to use the PPK kernel in a known kernel.

[0043] It is thus proposed to compute the distance between p and q as follows:

[0044] Equation (2)

[0045] This distance according to equation (2) can then be used in a known kernel function, where the individual inner products are computed with the PPK kernel. In case p and q are GMMs and GPs, the inner products required for this can be computed in an analytical way:

[0046] Equation (3)

[0047] where ; ; denote the weights, the mean values or the covariances of the mixture components k of the GMM or the support points of the GP, respectively.

[0048] Most stationary kernels have a so-called length scale or also called correlation parameter which scales the distance between two elements. This parameter is usually estimated numerically in order to optimally fit the data on which the GP is based. This can be performed for example with "Automatic Relevance Determination" (ARD) for short. In order to be able to use the correlation parameter, equation (3) can be extended:

[0049] Equation (4)

[0050] where the length scale for the kth mixture component and the support point.

[0051] The distance calculated according to equation (2) can then be used for example in a Matern 5 / 2 kernel. Other kernels are also conceivable, such as a Matern 3 / 2 kernel:

[0052] Equation (2)

[0053] Figure 1 A flowchart (2) of a method for optimizing a predefined policy of a robot is schematically shown, wherein the policy is a GMM.

[0054] The method starts with step S21. In this step, a GMM is provided, which has preferably been trained and is thus suitable for proposing sub-trajectories for a robot. Furthermore, a GP is initialized in step S21, so that the GP is suitable for outputting costs depending on the parameters of the GMM. In this case, the costs characterize an outlay, such as time or energy, which the robot has to raise in order to reach a target position from its starting position / state while adhering to the policy, in particular to a trajectory determined in accordance with the policy.

[0055] The GP comprises at least one kernel function k which is used to determine the distance between the GMM and the GP according to the calculation set out above.

[0056] This is followed by step S22. Here, a plurality of trajectories is created in accordance with the policy. The costs belonging to each of the created trajectories are determined. The costs can be determined for example experimentally, by manipulating the robot in accordance with the respective created policy.

[0057] This is followed by step S23. In this step, the GP is optimized by means of BO and in accordance with the used parameters of the GMM and the costs associated therewith, so that the GP predicts the costs depending on the parameters.

[0058] Step S24 can then optionally be performed. In this step, the GP is used to determine other trajectories and their costs in a targeted manner. These other trajectories serve to explore the search space and to approach the optimal cost. To this end, parameters for the GMM are suggested in accordance with the outputted uncertainty of the GP or in accordance with an assumed optimal value in terms of the optimal cost predicted by the GP, in order to then create further trajectories in accordance with the GMM parameterized with these parameters.

[0059] After the other trajectories and costs have been determined in step S24, the GP is then re-optimized in step S24 with the BO and the new data.

[0060] After step S23 or after step S24 is step S25. In step S25, the optimal parameters of the GMM belonging to the optimal cost outputted by the GP are then determined. This can be performed, for example, using the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method. This is followed by the replacement of the parameters of the GMM by the optimal parameters.

[0061] After the end of step S25, the method is complete. It should be noted that step S24 can be performed several times in succession. It is conceivable that after step S25 is a further step in which the robot is then operated using the adapted GMM from step S25. The use of the GMM for a robot or production machine or autonomous vehicle should be illustrated exemplary in the subsequent figures.

[0062] Figure 2 An executor 10 is shown. At preferably regular time intervals, the environment is detected in a sensor 30, in particular an imaging sensor, for example a video sensor, which can also be given by a plurality of sensors, for example a stereo camera. Other sensors are also conceivable, for example radar, ultrasound or lidar, but also a GPS or a rotation rate sensor, an accelerometer, an angle encoder at the executor itself. The sensor signal S of the sensor 30 (or each sensor signal S in the case of a plurality of sensors) is transmitted to a control system 40. The control system 40 thus receives a series of sensor signals S. From this, the control system 40 determines a maneuvering signal A, which is transmitted to the executor 10.

[0063] The detected sensor signals are then fed to the parameterized GMM after step S25.

[0064] The GMM then outputs a sub-trajectory. In accordance with the sub-trajectory, a maneuvering signal for the executor 10 is determined by means of a computing unit, which is correspondingly maneuvered and performs the corresponding action.

[0065] In other preferred embodiments, the control system 40 comprises a single or multiple processor 45 and at least one machine-readable storage medium 46 on which instructions are stored which, when executed on the processor 45, cause the control system 40 to perform a method according to the present application.

[0066] In alternative embodiments, instead of or in addition to the actuators 10, a display unit 10a is provided which can display the output of the GMM.

[0067] Figure 2 It is shown how the control system 40 can be used to control an at least partially autonomous robot, here an at least partially autonomous motor vehicle 100.

[0068] The actuators 10, which are preferably arranged in the motor vehicle 100, can for example be brakes, drive means or steering means of the motor vehicle 100.

[0069] Alternatively, the at least partially autonomous robot can also be another mobile robot (not depicted), for example a robot which advances by flying, swimming, diving or walking. The mobile robot can for example also be an at least partially autonomous lawnmower or an at least partially autonomous transport robot. Even in these cases, the control signals A can be determined such that the drive means and / or steering means of the mobile robot are controlled such that the at least partially autonomous robot prevents a collision, for example with an object identified by the artificial neural network 60.

[0070] Alternatively or additionally, the display unit 10a can be controlled in accordance with the control signals A. It can also be possible, for example in the case of a motor vehicle 100 with non-automated steering means, to control the display unit 10a with the control signals A such that the display unit outputs an optical or acoustic warning signal when it is determined that the motor vehicle 100 is in the vicinity.

[0071] Figure 3 An embodiment is shown in which the control system 40 is used to control a production machine 11 of a production system 200 in such a way that actuators 10 which control the production machine 11 are controlled. The production machine 11 can for example be a machine for punching, sawing, drilling and / or cutting.

[0072] The sensors 30 can then for example be optical sensors which for example detect properties of the production products 12a, 12b. It can be possible that these production products 12a, 12b are movable. It can be possible that the actuators 10 which control the production machine 11 are controlled in accordance with the allocation of the detected production products 12a, 12b in order for the production machine 11 to correspondingly perform a subsequent machining step of the correct production product among the production products 12a, 12b.

Claims

1. A method (20) for optimizing a predefined policy of a robot, wherein the policy is a Gaussian mixture model outputting at least one sub-trajectory depending on a start state and a goal state of the robot, the method comprising the steps of: initializing (S21) a Gaussian process adapted to estimate a cost the robot has to spend in order to reach the goal position depending on a parameterization of the Gaussian mixture model, wherein the Gaussian process has at least one kernel obtaining input arguments depending on a distance determined according to a probabilistic product kernel between probability distributions respectively characterized by the Gaussian mixture model and the Gaussian process; creating (S22) a plurality of trajectories depending on the policy; determining (S23) a cost for each of the plurality of trajectories; optimizing (S24) the Gaussian process such that the Gaussian process estimates the determined costs for the plurality of trajectories depending on used parameters of the Gaussian mixture model; determining an optimal parameter for the Gaussian mixture model by means of the Gaussian process such that the Gaussian process outputs an optimal cost for the optimal parameter; replacing (S25) parameters of the Gaussian mixture model by the optimal parameter.

2. The method according to claim 1, wherein the distance is calculated according to a probabilistic product kernel depending on an inner product between the Gaussian mixture model p and the Gaussian process.

3. The method according to claim 2, wherein the inner product is determined depending on a sum over a plurality of normal distributions, wherein the plurality of normal distributions respectively comprises a normal distribution for each possible combination between a mixture component of the Gaussian mixture model and a support point of the Gaussian process, wherein the plurality of normal distributions is respectively parameterized with an expectation value of a respective mixture component giving an expectation value of the respective support point and with a sum over variances of the respective mixture component and the respective support point, wherein the plurality of normal distributions is respectively weighted with weights of the respective mixture component of the Gaussian mixture model and the respective support point of the Gaussian process.

4. The method according to claim 1, wherein the distance is related to a root of an inner product of a difference between the Gaussian mixture model and the Gaussian process and the difference.

5. The method according to claim 4, wherein the inner product of the difference is solved according to a second binomial formula and each term of the solved second binomial formula is determined as a sum over a plurality of normal distributions, wherein the inner product between the Gaussian mixture model and the Gaussian process is determined depending on a sum over a plurality of normal distributions, and wherein the distance is calculated according to a probabilistic product kernel depending on the inner product between the Gaussian mixture model and the Gaussian process.

6. The method according to any of the preceding claims 1-5, wherein the kernel is a Matern kernel.

7. The method according to any of the preceding claims 1-5, wherein the kernel is a Matern 5 / 2 kernel.

8. The method according to any of the preceding claims 1-5, wherein the mixture components of the Gaussian mixture model and the support points of the Gaussian process are assigned a length scale, respectively, wherein the length scale scales the distance determined according to the probability product kernel.

9. A control system comprising a single or multiple processors (45) and at least one machine readable storage medium (46) on which instructions are stored which, when executed on the processors (45), cause the control system (40) to perform the method according to any of claims 1 to 8.

10. A computer program product comprising a computer program which, when executed on a processor (45), is arranged to perform the method according to any of claims 1 to 8.

11. A machine readable storage medium on which a computer program is stored which, when executed on a processor (45), is arranged to perform the method according to any of claims 1 to 8.

Images