Methods and systems for applying machine learning to applications

Abstract

A method for applying machine learning, in particular state-based machine learning, to an application, comprising the following steps: a) generating a candidate strategy (P) by a learner (20); b) executing a program (80) in at least one simulation application (51, 52) based on a candidate parameter set (PC1, PC2), wherein the candidate parameter set (PC1, PC2) is provided based on the candidate strategy (P) and the state (S) of the simulation application (51, 52), and the execution of the program provides an intermediate result (R1, R2) of the measured set (PC1, PC2) of the candidate parameters based on measured performance information of the execution of the program (80); c) collecting a predetermined number of intermediate results (R1, R2) by a trainer (30), and providing a final result (R) based on a combination of the candidate parameters (PC) and / or the state (S) and the measured performance information; and d) generating a new candidate strategy (P) by the learner (20) based on the final result (R).

Description

[0001] The present disclosure relates to methods for applying machine learning to applications, systems for applying machine learning to applications, parameter sets, and program elements.

[0002] The tuning of parameters for applications (particularly industrial applications, such as robotic applications) is a tedious and time-consuming task, and is usually performed during commissioning and often delays the start of production. This task is often made even more difficult because the correct choice of parameters depends on the varying conditions of the application (e.g. handle material).

[0003] Machine learning can provide improvements. While the amount of data required to learn a given task depends on the complexity of the task and the complexity of the learning algorithm, effective generalization of machine learning usually requires a dataset with large and diverse samples. In practice, in order to validate the model, some data needs to be retained for testing purposes. However, in many industrial applications, it is often difficult to obtain a sufficient amount of quality data for machine learning. Learning methods need to be able to cope with this necessity.

[0004] A common approach is so-called exploratory learning. In this learning method, the algorithm searches for new possibilities and alternatives to adapt to changes in the environment through continuous experimental activities. However, exploratory learning is relatively expensive, dangerous, and very often not even possible on real hardware.

[0005] For example, machine learning has recently been applied to robotics applications to learn basic movements. Thus, the machine learning algorithm specifically modifies the program executed in the robotics application in order to improve the task to be solved.

[0006] Instead of applying machine learning to real industrial applications, simulation can be used in order to perform machine learning. Usually, it is quite difficult to transfer the results of simulation-based machine learning (so-called offline machine learning) to real applications. Usually, simulation is not sufficiently adapted to reality. The level of learning (e.g. controller, motion, task) must be carefully selected to prevent the reality gap (the reality gap is the gap between the simulated application and the real application) with serious consequences. In addition, in most cases, the time required for the learning process is usually too long for practical applications. In addition, the definition of an optimization process for parameters with complete definition of constraints is extremely difficult.

[0007] Therefore, there is a need for improved methods for applying machine learning to applications.

[0008] According to aspects of the present disclosure, there is provided a method for applying machine learning (particularly state-based machine learning) to an application, comprising the following steps:

[0009] a) The learner generates candidate strategies;

[0010] b) executing a program in at least one simulation application based on a set of candidate parameters, the set of candidate parameters being provided based on a candidate strategy and a state of the simulation application, the execution of the program providing intermediate results of the measured set of candidate parameters based on measured performance information of the execution of the program;

[0011] c) collecting a predetermined number of intermediate results by the trainer and providing a final result based on a combination of candidate parameters and / or states and measured performance information; and

[0012] d) The learner generates new candidate strategies based on the final results.

[0013] Preferably, the simulation application starts executing the program. After a period of time, the simulation application needs a candidate parameter set to further execute the program. The simulation application then specifically requests the candidate parameter set together with the state of the simulation application. Based on the state, the simulation application is provided with a candidate parameter set mapped to the state.

[0014] The term "candidate strategy" describes the mapping of states to candidate parameter sets.

[0015] The term "state" describes the characteristics of an application (eg in case of a real robotics application, where an object should be inserted into a pocket), the state might be information about the type of object to be inserted.

[0016] The simulated application is preferably a simulation of a real application. Further preferably, the simulated application is a simulation of a real industrial application (particularly a real robotic application) executed on a machine (particularly a robot).

[0017] The simulation application is preferably executed on the simulation model.The simulation application preferably further comprises an application program and an application control.

[0018] For example, when the real application is a real robotic application, the simulation application preferably includes a robotic application program and a robot control, including the interaction between the robot and robot environment objects.

[0019] By using simulation applications (particularly distributed simulation ensembles) to supplement machine learning based on real applications, an increased amount of learning data necessary for correct machine learning can be obtained. Thus, the speed of the learning process can be increased.

[0020] In order to learn, a large number of experiments need to be run. If this were to be run on a physical robot, it would require a lot of time that is not available, and if it were possible to run the experiments in parallel on more than one physical robot, it would require many robots that are not available. Therefore, a benefit of simulation-based learning is the ability to run a large number of experiments without requiring either (1) many physical robots or (2) a lot of time.

[0021] The learner is preferably configured to generate an initial mapping application state (referred to as a strategy) and a set of candidate parameters. In addition, the learner is configured to generate new candidate strategies based on performance information of the application. The learner preferably includes a machine learning backend. In an advantageous embodiment, the learner uses an appropriate learning algorithm for generating candidate parameters. Preferably, the learning algorithm is any kind of meta-inspiration for optimization, such as a genetic algorithm, taboo search, or simulated annealing. Suitable algorithms for this type of learning are learning with a state table, particularly for a small number of states, or artificial neural networks, particularly for a large number of states.

[0022] The performance information preferably includes different elements, such as failure, success, probability of success, cycle time, quality, quality metrics, etc.

[0023] It is not required that the feasibility of candidate parameter sets be predefined. Preferably, errors in execution caused by infeasible sets of candidate parameters will be flagged by performance information. In the event that a program in the simulation application is based on an infeasible set of candidate parameters, the simulation is reset and the learning process continues with the next set of candidate parameters.

[0024] Preferably, the set of candidate parameters includes parameters for functionality of the program related to movement and / or tasks of the machine executing the program (e.g. such as related to robotic picking up objects), search radius and / or search pattern for objects and / or increments describing the granularity of the search.

[0025] For real applications characterized by the maximum improvement of its performance with respect to predetermined criteria, the final result is preferably the final strategy, and thus the final set of parameters based on the state of the application. For example, the predetermined criteria in a real robotics application for searching and picking up objects would include search time and probability of success.

[0026] In a preferred embodiment, the program is executed in at least one simulation application and in at least one real application (particularly a real robot application) preferably executed on a machine (particularly on a robot). Preferably, the program is executed simultaneously in multiple simulation applications and in at least one real application.

[0027] Preferably, the real application starts executing the program. After a period of time, the simulation application needs a candidate parameter set to further execute the program. The real application then requests the candidate parameter set together with the state of the real application. Based on the state, the real application is provided with a candidate parameter set mapped to the state.

[0028] In another preferred embodiment, the method comprises starting the learning in simulation and then transferring the results to a real application, in particular a real robotic application.

[0029] The real application is preferably executed on a machine, further preferably the real robot application is executed on a robot.

[0030] Real robotic applications preferably include robotic applications with mechanical manipulators, robotic controls and robotic applications as well as non-robotic elements (eg, programmable logic controllers (PLCs), material handling, machine processing centers, etc.).

[0031] Executing programs simultaneously in simulated applications and real applications reduces the so-called reality gap. Simulated applications are almost never an exact mirror image of real applications. Therefore, when the same set of candidate parameters is used to execute the corresponding program, the final result of the simulated application is usually different from the final result of the real application. When machine learning is applied only to simulated applications, there is an increased possibility of the development of machine learning algorithms in a certain direction, which improves the final result of the simulated application, but does not improve the final result of the real application in the same way. In this sense, there is a gap between simulation and reality, which is called the reality gap. This phenomenon can lead to the final result proposed by machine learning based only on simulated applications, which actually seems to be the best final result, but it may not be the best final result considering the real application.

[0032] Simultaneously in this case has to be understood as simultaneous taking into account the execution of one or more complete sets of candidate parameters. For example, a program in a simulation application need not be executed in exactly the same time as a program in a real robot application.

[0033] Thus, during the runtime of the entire learning method, there can be times when the program is executed only in the simulation application or only in the real application. Preferably, when exploratory learning is applied to the simulation application, the program is executed only in the simulation application. When the exploratory learning is completed, the program is also executed again in the real application. Further preferably, the program is executed in parallel in the simulation application and in the real application complementarily and / or redundantly.

[0034] Execute the same program in one or more simulated applications as well as in the real application.

[0035] Preferably, the actual application is also performed based on the candidate parameter set and an additional machine learning process is initiated, further preferably using an on-board algorithm, separate from the method and thus without the need for systematic application of the method.

[0036] In a preferred embodiment, a program is executed in cooperating real applications, wherein the candidate policy defines the assignment of tasks to each of the cooperating real applications.

[0037] A special case of learning is learning the assignment of tasks to several machines (especially robots) in a cell. A candidate strategy defines which subtask an individual machine performs, e.g. which parts it assembles according to candidate parameters "object type" or "pick location" when associated with several robotic arms.

[0038] In a preferred embodiment, steps a) to d) are repeated until a stop criterion is met. Preferably, the stop criterion includes the execution volume of the program and / or target measurement performance information.

[0039] In a preferred embodiment, the candidate parameter set has a parameter range, wherein the candidate parameter set executed on the simulated application has a wider parameter range than the candidate parameter set executed on the real application.

[0040] A wider parameter range for the candidate parameters increases the risk of unsafe parameterization. For example, unsafe parameterization may result in damage to the real application when a program running with unsafe parameters instructs the real application to damage itself or the environment. In the case of a real robot application, the robot may collide with surrounding objects when it runs with unsafe parameters. Therefore, it is preferred to test those situations in a simulation application. Based on a set of candidate parameters with a wide parameter range, a subset of candidate parameters with a narrower range can be learned or determined, wherein in the simulation application, the subset of candidate parameters with a narrower range has been determined to be safe. Preferably, if a certain range of candidate parameters is determined to be safe, the information is based on measured performance information.

[0041] In a preferred embodiment, real data of the machine is acquired while the program is executed in the real application, wherein the simulation application is modified based on the real data.

[0042] Preferably, the real data is acquired using measurements and additional estimates from the real world, in particular physical parameters such as weight, friction, coefficients, etc. Based on the real information, the simulation application can be modified in order to reduce the reality gap, which describes the difference between the results of the simulation application and the results of the real application when the corresponding program is executed with the same candidate parameters.

[0043] In a preferred embodiment, the candidate parameter set is assigned to the simulation application and / or the real application by at least one manager based on the candidate strategy and state. Preferably, the candidate parameter set is requested by the simulation application and / or the real application. Preferably, the intermediate results of the measured set of candidate parameters are received by the manager based on the measured performance of the execution of the program.

[0044] The manager preferably triggers the execution of the application program.Further preferably, the manager collects or receives performance information of the application, and preferably provides the performance information to the trainer.

[0045] Preferably, when the simulation application and / or the real application requests a candidate parameter set, the manager provides the candidate parameter set for the simulation application and / or the real application based on the state. The state of the simulation application and / or the real robot application is thus provided by the simulation application and / or the real robot application, or provided by another source, which is different from the simulation application and / or the real application, such as a camera, a URL, a file system and / or an API endpoint. In the second case, the real application and / or the simulation application will only provide information about the candidate parameter set for which it needs the current state at this time.

[0046] In an advantageous embodiment, a separate application manager is associated to each application, ie, simulated application or real application.

[0047] In a preferred embodiment, the intermediate results are received by a trainer, wherein the trainer triggers the generation of a new policy by the learner.

[0048] The trainer preferably requests and receives initial candidate policies and / or new policies from the learner. In addition, the trainer preferably assigns candidate policies to managers. If there are several managers, the trainer assigns candidate policies to individual managers. In addition, the trainer preferably receives performance information from one or more managers and triggers the generation of new candidate policies by the learner. The trainer preferably receives new candidate policies from the learner.

[0049] According to aspects of the present disclosure, a system for applying machine learning (particularly state-based machine learning) to an application includes: a learner configured to generate a learning strategy; at least one simulation application; a program configured to execute in the simulation application based on a set of candidate parameters, the set of candidate parameters being provided based on the candidate strategy and the state of the simulation application. The program is configured to provide intermediate results of a measured set of candidate parameters based on measured performance information of the execution of the program. The system further includes a trainer configured to collect a predetermined number of intermediate results, wherein the trainer is configured to provide a final result based on a combination of the candidate parameters and / or state and the measured performance information. The learner is configured to generate a new candidate strategy based on the final result.

[0050] In a preferred embodiment, the system comprises at least one real application, wherein the program is configured to be executed in a simulated application and in a real application based on the candidate parameter set.Preferably, the program is configured to be executed in multiple simulated applications and in at least one real application simultaneously.

[0051] In a preferred embodiment, the candidate parameter set has a parameter range, wherein the candidate parameter set executed on the simulated application has a wider parameter range than the candidate parameter set executed on the real application.

[0052] In a preferred embodiment, the system includes a sensor configured to acquire real data while executing a program in a real application, wherein the simulation application is configured to be modified based on the real data.

[0053] In a preferred embodiment, the system comprises a manager configured to assign a candidate parameter set to a simulation application and / or a real application based on a candidate strategy and a state. Preferably, the manager is configured to receive intermediate results of the measured set of candidate parameters based on measured performance of execution of the program, wherein further preferably, the learner is configured to receive the intermediate results through the trainer, and the trainer is configured to trigger the generation of a new strategy by the learner.

[0054] Preferably, one or more of the real application, the simulation application, the learner, the manager and the trainer are combined in a single software or hardware component. In particular, the learner and the manager (in particular, multiple managers) are combined in a single software or hardware component.

[0055] According to aspects of the present disclosure, a final parameter set is obtained through the method.

[0056] According to an aspect of the present disclosure, a program element is provided, which, when executed on a system for applying machine learning (particularly state-based machine learning) to an application, instructs the system to perform the following steps:

[0057] a) The learner generates candidate strategies;

[0058] b) executing a program in at least one simulation application based on a set of candidate parameters, the set of candidate parameters being provided based on a candidate strategy and a state of the simulation application, the execution of the program providing intermediate results of the measured set of candidate parameters based on measured performance information of the execution of the program;

[0059] c) collecting a predetermined number of intermediate results by the trainer and providing a final result based on a combination of candidate parameters and / or states and measured performance information; and

[0060] d) The learner generates new candidate strategies based on the final results.

[0061] According to an aspect of the present disclosure, a computer-readable medium is provided on which the program element is stored.

[0062] These and other features, aspects and advantages of the present invention will become better understood with reference to the accompanying drawings and the following description.

[0063] Figure 1 A system for applying machine learning to an application is shown.

[0064] Figure 2 A real robot application according to the first embodiment is shown.

[0065] Figure 3 Show Figure 2 A zoomed-in view of the search pattern.

[0066] Figure 4 A real robot application according to the second embodiment is shown.

[0067] Figure 5 Demonstrates methods for applying machine learning to applications. DETAILED DESCRIPTION

[0068] Figure 1 The system 10 shown in the figure relates to a system 10 for applying machine learning to a real application (particularly a real robot application 71) of a machine (particularly a robot 70). The system 10 includes a learner 20, a trainer 30, a first manager 41, a second manager 42, a third manager 43, a first simulation application 51, a second simulation application 52, a simulation model 60, a real robot application 71, and one or more robot programs 80.

[0069] The first simulation application 51 runs on a first virtual machine or computer device, and the second simulation application 52 runs on a second virtual machine or computer device. The first simulation application 51 and the second simulation application 52 can also run on the same virtual machine and / or the same computer device. The first simulation application 51 runs on a first simulation 53, and the second simulation application 52 runs on a second simulation 54. The first simulation 53 and the second simulation 54 are based on a simulation model 60, which provides a platform for executing a robot program 80 in the first simulation application 51 and the second simulation application 52.

[0070] A real robot application 71 runs on a robot 70. The robot 70 is a real physical machine. The simulation model 60 is a digital reflection of the robot 70. Ideally, a robot program 80 executed on the first simulation 53 or the second simulation 54 results in the same intermediate results as a robot program 80 executed on the robot 70. However, there are almost always differences between simulation and physical machines.

[0071] In this example, the robot program 80 can be expressed by the following pseudo code:

[0072] float r / / search radius

[0073] float I / / search increment

[0074] int p / / search mode

[0075] Pick (Pos_0)

[0076] Move (Pos_1)

[0077] Insert_object (r, i, p).

[0078] The robot program 80 contains instructions for picking up an object at position Pos_0, moving it to position Pos_1, and inserting it into a pocket 94 or similar near Pos_1. In this example, the insertion method has three parameters. The first parameter r is the radius around Pos_1 where the robot will search for the pocket, where the radius is the search radius of the robot 70. The second parameter p is the search mode. The parameter p determines, for example, whether the search mode is radial with rotation or radial without rotation. The third parameter i is the increment, which defines the granularity of the search.

[0079] In order to apply machine learning to the first simulation application 51, the second simulation application 52, and the real robot application 71, the learner 20 is configured to generate an initial candidate strategy P. In addition, the learner 20 is configured to generate a new candidate strategy P. The candidate strategy P is a mapping of states S1, S2, S3 to candidate parameter sets PC1, PC2, PC3, which includes a first parameter r, a second parameter p, and a second parameter i.

[0080] The learner 20 is connected to the trainer 30, which is itself connected to a first manager 41, a second manager 42 and a third manager 43. Although it is possible to have only a single manager, in this example one manager 41, 42, 43 is associated with one application 51, 52, 71 respectively. The first manager 41 is associated with the first simulation application 51, the second manager 42 is associated with the second simulation application 52, and the third manager is associated with the real robot application 71.

[0081] In general, the learner 20 provides a plurality of candidate strategies P to the trainer 30. The trainer 30 provides one or more candidate strategies P to the first simulation application 51, the second simulation application 52, and the real robot application 71. After starting the corresponding program 80, the first simulation application 51 requests the first set PC1 of candidate parameters from the first manager 41 together with the first state S1 providing the first simulation application 51. Correspondingly, the second simulation application 52 requests the second set PC2 of candidate parameters from the second manager 42 together with the second state S2 providing the second simulation application 52. Further correspondingly, the real robot application 71 requests the set of candidate parameters from the third manager 43 together with the third state S3 providing the real robot application 71. The first state S1, the second state S2, and the third state S3 in this case describe which objects 91, 92, 93 should be inserted into the hole 94, such as a cubic object, a cylindrical object, etc.

[0082] The first manager 41 selects the relevant candidate strategy P according to the provided first state S1, and provides the first set PC1 of candidate parameters for the first simulation application 51. The second manager 42 selects the relevant candidate strategy P according to the provided second state S2, and provides the second set PC2 of candidate parameters for the second simulation application 52. The third manager 43 selects the relevant candidate strategy P according to the provided third state S2, and provides the third set PC3 of candidate parameters for the real robot application 71.

[0083] Via the corresponding first manager 41, second manager 42 and third manager 43, the first simulation application 51 provides the first intermediate result R1 to the trainer 30, the second simulation application 52 provides the second intermediate result R2 to the trainer 30, and the real robot application 71 provides the third intermediate result R3 to the trainer 30. The trainer 30 then provides the final result R to the learner 20, on which a new policy P is generated, until the machine learning is completed and the final policy set PF is found, which is then provided to the real robot application 71.

[0084] Figure 2 A real robot application 71 according to a first embodiment is shown. The robot 70 is controlled by a computer device 81. On the computer device 81, a program 80 is executed. The robot program 80 is provided with, for example, a third set PC3 of candidate parameters, which includes a first parameter r, a second parameter p and a third parameter i. Figure 3 As shown in FIG. 1 , the first parameter r determines the radius of the area for the search movement of the robot 70. The second parameter p, which sets the search mode, is set to a spiral search mode in this example. The third parameter i determines the increment for the search movement, in this example a spiral search movement.

[0085] As possible objects, there are a first object 91, a second object 92 and a third object 93, all of which are different from each other. The state of the real robot application 71 is defined by the object to be inserted. Therefore, there are three possible states related to the object to be inserted. When the first object 91 should be inserted, the state of the real robot application 71 is the first state S1. Correspondingly, when the second object 92 should be inserted, the state of the real robot application 71 is the second state S2. Further correspondingly, when the third object 93 should be inserted, the state of the real robot application 71 is the third state S3. In this example, the third object 93 should be inserted, which causes the third state S3.

[0086] The program simulation had an uncertainty of 100 random starting positions for each set of candidate parameters.

[0087] As learning algorithm, any kind of meta-heuristic for optimization (eg genetic algorithm, tabu search or simulated annealing) can be used to optimize the parameters.

[0088] In this example, the parameters are initialized with an approximate range of the first parameter r in the range of 3.0 mm and 20.0 mm, and a third parameter i in the range of 0.1 mm and 5.0 mm. In this example, the second parameter p is the stable spiral mode.

[0089] The real robot application 71 returns a third intermediate result R3, which in this example is the time required to complete the search.

[0090] Figure 4 A real robot application 171 according to a second embodiment is shown.

[0091] The real robot application 171 includes a robot 170 which picks up a fourth object 191 .

[0092] The camera device 173 makes a picture of the fourth object 191 in front of the robot 170 for determining the state of the real robot application 171. The camera device 173 generates a black and white bitmap 174, in particular of size 250×250. In contrast to the real robot application 71 according to the first embodiment, here the candidate parameter set PC that should be learned for the program 80 is two points with which the robot gripper 172 should come into contact when picking up the fourth object 191. In this example, the state of the real robot application 171 also involves the type of object to be picked up. Preferably, the gripping points are defined in the 3D space [RxRxR], and the entire black and white bitmap ({0,1} 250x250 ) to learn the optimal mapping in the parameter space [RxRxR]x[RxRxR].

[0093] exist Figure 5, a method for applying state-based machine learning to an application is illustrated in FIG. 5 , wherein in an exemplary embodiment, the real application is a real robot application 71 , and the machine executing the real robot application 71 is a robot 70 .

[0094] In step S0, the learner 20 generates one or more initial candidate policies P. A policy is a mapping of states to candidate parameters PC.

[0095] In step S1 , the learner 20 makes one or more policies P available to the trainer 30 .

[0096] In step S2 , the trainer 30 assigns one or more policies P to managers, in this example, the first manager 41 , the second manager 42 and the third manager 43 .

[0097] In step S3, the corresponding first manager 41, the second manager 42 and the third manager 43 respectively set the first set PC1 of candidate parameters, the second set PC2 of candidate parameters and / or the third set PC3 of candidate parameters of the robot program 80 based on the strategy P in the first simulation application 51, the second simulation application 52 and / or the real robot application 71.

[0098] In step S4 , the first simulation application 51 , the second simulation application 52 and / or the real robot application 71 are executed.

[0099] In step S5, the robot program 80 at a certain position in the program requires parameters. The first simulation application 51 requests a first set PC1 of candidate parameters while providing a first state S1. The second simulation application 52 requests a second set PC2 of candidate parameters while providing a second state S2. The real robot application 71 requests a third set PC3 of candidate parameters while providing a third state S3. The first state S1, the second state S2 and / or the third state S3 relate to the type of the inserted object.

[0100] In step S6, first manager 41 provides a first set PC1 of candidate parameters from candidate strategy P based on first state S1. Second manager 42 provides a second set PC2 of candidate parameters from candidate strategy P based on second state S2. Third manager 43 provides a third set PC3 of candidate parameters from candidate strategy P based on third state S3.

[0101] In step S7, the first simulation application 51, the second simulation application and / or the real robot application 71 continue the execution of the robot program 80, and each provides performance-related information, which preferably has candidate parameter values ​​and corresponding states, in the form of a first intermediate result R1, a second intermediate result R2 and / or a third intermediate result R3, respectively.

[0102] In step S8 , the first manager 41 , the second manager 42 and / or the third manager 43 make the first intermediate result R1 , the second intermediate result R2 and / or the third intermediate result R3 available to the trainer 30 .

[0103] In step S9, steps S4 to S7 are repeated. In an alternative embodiment, the managers 41, 42, 43 may collect the intermediate results R1, R2, R3 before making them available to the trainer 30. Alternatively, if the underlying states S1, S2, S3 are known a priori, multiple sets PC of candidate parameters may also be immediately available, and the execution of the corresponding application 51, 52, 71 obtains the corresponding set PC of candidate parameters according to the states S1, S2, S3. In both cases, the learner 20 uses the final results R associated with the states S1, S2, S3 to improve the policy P.

[0104] In step S10, the trainer 30 makes the final result R available to the learner 20. The trainer may also collect the intermediate results R1, R2, R3 from the managers 41, 42, 43 before sharing them with the learner 20. Alternatively, a database may be used to store the final result R and the learner 20 reads from it.

[0105] In step S11 , the learner 20 generates a new policy P based on the final result R.

[0106] After step 9, the process continues repeatedly from step 1 until a stopping criterion is met, such as the number of tested configurations or convergence of a stable solution. After this stopping criterion is met, the best learned solution is set as the final solution in the robot program 80 by providing the robot program 80 with the final policy PF in step 12.

[0107] After this step 12, no learning infrastructure is required for the robot to operate.

[0108] Reference Symbols List

[0109] 10 System

[0110] 20 Learner

[0111] 30 Trainer

[0112] 41 First Manager

[0113] 42 Second Manager

[0114] 43 Third Manager

[0115] 51 First Simulation Application

[0116] 52 Second Simulation Application

[0117] 53 First Simulation

[0118] 54 Second Simulation

[0119] 60 Simulation Models

[0120] 70 Machine (Robot)

[0121] 71 Real Applications (Real Robot Applications)

[0122] 80 Program (Robot Program)

[0123] 81 Computer device

[0124] 91 First Object

[0125] 92 Second object

[0126] 93 Third Object

[0127] 94 holes

[0128] 170 Machine (Robot)

[0129] 171 Real Applications (Real Robot Applications)

[0130] 172 Robot Gripper

[0131] 173 Photographic Device

[0132] 174 Bitmap

[0133] 191 The fourth object

[0134] PC candidate parameter set

[0135] The first set of candidate parameters for PC1

[0136] PC2 Second set of candidate parameters

[0137] PC3 The third set of candidate parameters

[0138] R Final Result

[0139] R1 First intermediate result

[0140] R2 Second intermediate result

[0141] R3 Third intermediate result

[0142] P Candidate Strategy

[0143] PF Final Strategy.

Claims

1. A method for applying machine learning to an application, comprising the steps of: a) generating a candidate policy (P) by a learner (20), wherein the candidate policy (P) describes a mapping of a state (S) to a candidate parameter set (PC1, PC2); b) allocating, by at least one manager (41, 42), the candidate parameter set (PC1, PC2) to at least one simulation application (51, 52) based on the candidate strategy (P) and the state (S); c) executing a program (80) in said at least one simulation application (51, 52) based on a candidate parameter set (PC1, PC2), said candidate parameter set (PC1, PC2) being provided based on said candidate strategy (P) and a state (S) of said simulation application (51, 52), said execution of said program providing intermediate results (R1, R2) of a measured set (PC1, PC2) of candidate parameters based on measured performance information of said execution of said program (80), wherein said candidate parameter set (PC1, PC2) is requested by said simulation application (51, 52) together with providing the state (S) of said simulation application (51, 52); d) collecting a predetermined number of intermediate results (R1, R2) by a trainer (30) and providing a final result (R) based on a combination of the candidate parameters (PC) and / or the state (S) and the measured performance information; and e) generating, by the learner (20), a new candidate strategy (P) based on the final result (R) using a machine learning algorithm, wherein the program (80) is executed in at least one simulation application (51, 52) and in at least one real application (71) executed on a machine (70); in acquiring real data of the machine (70) while executing the program (80) in the real application (71); Wherein the simulation application (51, 52) is modified based on the real data.

2. The method of claim 1, wherein The program (80) is executed in a real robot application executed on the robot.

3. The method as claimed in claim 1, wherein the program (80) is executed simultaneously in a plurality of simulation applications (51, 52) and in at least one real application (71).

4. The method of claim 1, wherein executing said program (80) in a collaborative real application (71), Wherein the candidate strategy (P) defines the assignment of tasks to each of the cooperating real applications (71).

5. The method according to any one of claims 1 to 4, wherein Repeat steps a) to d) until the stopping criterion is met. Preferably, the stopping criterion includes the execution volume of the program (80) and / or target measurement performance.

6. The method according to any one of claims 1 to 4, wherein The candidate parameter set (PC) has a parameter range; The candidate parameter set (PC1, PC2) executed on the simulation application (51, 52) has a wider parameter range than the candidate parameter set (PC3) executed on the real application (71).

7. The method according to any one of claims 1 to 4, comprising the steps of: At least one manager (41, 42, 43) assigns the candidate parameter set (PC1, PC2, PC3) to the simulation application (51, 52) and / or the real application (71) based on the candidate strategy (P) and the state (S); The candidate parameter set (PC1, PC2, PC3) is requested by the simulation application (51, 52) and / or the real application (71); as well as The intermediate results (R1, R2, R3) of a measured set (PC) of candidate parameters are received by the manager (41, 42, 43) based on measured performance of the execution of the program (80).

8. The method according to any one of claims 1 to 4, comprising the steps of: The trainer (30) receives the intermediate results (R1, R2, R3); The trainer (30) triggers the generation of a new policy (P) by the learner (20).

9. A system (10) for applying machine learning to an application, comprising: A learner (20) configured to generate a candidate policy (P), wherein the candidate policy (P) describes a mapping of a state (S) to a candidate set of parameters (PC1, PC2); at least one simulation application (51, 52); A manager (41, 42) configured to assign the candidate parameter set (PC1, PC2) to the at least one simulation application (51, 52) based on the candidate strategy (P) and the state (S); a program (80) configured to be executed in the simulation application (51, 52) based on a candidate parameter set (PC), the candidate parameter set (PC) being provided based on a candidate strategy (P) and a state (S) of the simulation application (51, 52), wherein the program (80) is configured to provide an intermediate result (R1, R2) of a measured set (PC1, PC2) of candidate parameters based on measured performance information of the execution of the program (80), wherein the candidate parameter set (PC1, PC2) is requested by the simulation application (51, 52) together with providing the state (S) of the simulation application (51, 52); and a trainer (30) configured to collect a predetermined number of intermediate results (R1, R2), wherein the trainer (30) is configured to provide a final result (R) based on a combination of the candidate parameters (PC) and / or the state (S) and the measured performance information; wherein the learner (20) is configured to generate new candidate strategies (P) based on the final result (R) using a machine learning algorithm, the system further comprising: at least one real application (71); wherein the program (80) is configured to be executed in the simulation application (51, 52) and in the real application (71) based on the candidate parameter set (PC); a sensor configured to acquire real data while executing the program (80) in the real application (71); Wherein the simulation application (51, 52) is configured to be modified based on the real data.

10. The system of claim 9, The program (80) is configured to be executed simultaneously in a plurality of simulation applications (51, 52) and in at least one real application (71).

11. The system of claim 9, wherein The candidate parameter set (PC) has a parameter range; wherein The candidate parameter set (PC1, PC2) executed on the simulation application (51, 52) has a wider parameter range than the candidate parameter set (PC3) executed on the real application (71).

12. The system of any one of claims 9 to 11, comprising: A manager (41, 42, 43), the manager (41, 42, 43) being configured to assign the candidate parameter set (PC1, PC2, PC3) to the simulation application (51, 52) and / or the real application (71) based on the candidate strategy (P) and the state (S).

13. The system of any one of claims 9 to 11, comprising: A manager (41, 42, 43), the manager (41, 42, 43) being configured to receive the intermediate results of the measured set (PC1, PC2, PC3) of candidate parameters based on the measured performance of the execution of the program (80), wherein the learner (20) is configured to receive the intermediate results (R1, R2, R3) via a trainer (30), and the trainer (30) is configured to trigger the generation of a new strategy by the learner (20).

14. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method of any one of claims 1 to 8.

Images