Apparatus and method for planning operation of a technical system

Abstract

The invention relates to a method, computer program and machine readable storage medium for planning the operation of a technical system in its environment. A computer implemented method for planning the operation of a technical system in its environment. The method comprises the steps of: i) obtaining state information comprising: a current domain, a time step and a current state; ii) determining the cost of reachable states from the current state by a heuristic method; iii) selecting a heuristic method from a predefined set of heuristic methods according to a policy depending on the state information and the cost; iv) selecting a state from the reachable states with the lowest cost returned by the selected heuristic method, and v) determining the operation of the technical system from the set of possible operations that the technical system has to perform to reach said state with the lowest cost returned by the selected heuristic method.

Description

TECHNICAL FIELD

[0001] The present invention relates to a method, computer program and machine readable storage medium for planning the operation of a technical system in its environment. BACKGROUND

[0002] A heuristic is a method of solving a problem that is derived from previous experience with similar problems. Heuristic techniques are not guaranteed to be optimal, but are sufficient to achieve an immediate goal. Such techniques are often used in computer science to quickly find an approximate solution to a problem when a classical approach is slow or unable to find any exact solution, for example, finding a path for a robot to reach an immediate goal. For example, heuristics are often used to improve the convergence of search algorithms such as A* search by deciding which possibilities to explore first.

[0003] Heuristic forward search is one of the most popular and successful techniques in classical planning. Despite the large number of heuristics, it is well known that the performance (i.e. the amount of information) of heuristics differs between different domains. While in optimal planning it is easy to combine multiple admissible heuristic estimates (e.g. using the maximum), in satisficing planning it is difficult to combine inadmissible heuristic estimates in a reasonable way. The reason for this is that when all estimates are aggregated, highly inaccurate and uninformative heuristic estimates can negatively affect the overall search process.

[0004] Advantages of the invention

[0005] Since the performance of heuristics differs between different domains, alternating between multiple heuristics during the search makes it possible to use all heuristics equally and improves performance.

[0006] However, this approach ignores the internal search dynamics of the planning system, which can help to select the most useful heuristic for the current expansion step.

[0007] The inventors propose to exploit a strategy, preferably trained via Dynamic Algorithm Configuration (DAC), for dynamic heuristic selection that takes into account the internal search dynamics of the planning system. This has the advantage that it generalizes over existing approaches and can improve the performance of heuristic search exponentially and outperforms existing approaches in terms of coverage. Thus, the invention finds a solution faster than classical approaches and the solution is very close to the optimal solution.

[0008] DAC formulates the dynamic algorithm configuration as a contextual Markov decision process to leverage reinforcement learning in a data-driven manner to learn a policy for online adjustment of algorithm parameters, such that reinforcement learning learns a policy not only for a single instance, but also across a set of instances. DAC is disclosed in detail here: https: / / ml.informatik.uni-freiburg.de / papers / 20-ECAI-DAC.pdf. SUMMARY

[0009] In a first aspect, a computer-implemented method for planning operations of a technical system within its environment is disclosed. The environment is characterized by a current domain of a set of different domains, a current state of a set of states of each domain, and a set of possible operations that can be performed by the technical system within each domain. The method comprises the following steps:

[0010] i) obtaining state information comprising the current domain, a time step, and the current state of the environment.

[0011] ii) determining a cost of a plurality of reachable states from the current state by each heuristic method of a set of predefined heuristic methods;

[0012] iii) selecting one heuristic method from the set of predefined heuristic methods by a policy depending on the state information and in particular the determined costs. The heuristic method is configured to estimate a cost of reaching a goal state from a given state, and the policy has been trained to select the heuristic method in such a way that the technical system reaches the goal state with least search effort at the planning time by performing the operation;

[0013] iv) selecting a state from the reachable states with the lowest cost determined by the selected heuristic method, which is selected according to the policy; and

[0014] v) determining an operation of the technical system from the set of possible operations that the technical system has to perform to reach the state with the lowest cost.

[0015] It is proposed that after the step of determining an operation of the technical system by selecting an operation with the lowest value returned by the selected heuristic method depending on the current state, a control signal can be determined that is configured to control the technical system to perform the next operation. Still further, in a subsequent step, the technical system can be controlled depending on the determined operation or depending on the determined control signal.

[0016] The method has the advantage that the policy, in particular with its special input: the state information, and in particular the costs, can guide the technical system more effectively towards the target state. For example, if the technical system typically advances towards an obstacle due to a standard heuristic planning, the technical system needs some way to find a way around the obstacle. Due to the policy, a heuristic method is chosen that minimizes the total number of states required to reach the target state, which will select a heuristic method that can prevent the technical system from going around in circles by flexibly selecting from a set of heuristic methods the one that has the expected minimum number of expanded states to reach the target state.

[0017] Another advantage is that the planning process can be used for a wide variety of different environments and is thus flexible. The planner can switch between domains. Further, due to the policy, the optimal heuristic method can be found for different progress of the technical system towards reaching its target state, e.g. during the beginning a coarse heuristic method is more effective, where near the target state a more fine-grained heuristic method is more effective. Thus, the planning method is very powerful and flexible.

[0018] In the domains, the structure and / or classification of the environment of the technical system can be understood, e.g. indoor / outdoor or city / highway, etc. In the domains, also the environmental conditions of the technical system can be understood, e.g. weather conditions (rainy, sunny, snowy, etc.) or light conditions, etc.

[0019] The current state can also be extended and include features characterizing the internal state of the technical system.

[0020] Note that the method can be applied directly to a setting comprising multiple target states.

[0021] Preferably, the state information can also include the costs of the previously determined previous states, and the policy also depends on the previously determined costs to select the heuristic method.

[0022] Preferably, the policy can be configured or trained to select at each step of the planning process the heuristic method that provides the most information. A cost has to be spent to perform an operation of the technical system. The total cost refers to the sum of the necessary costs to reach the target state. For example, the cost can correspond to energy and / or time.

[0023] The reachable states are those states that the technical system can reach immediately by performing at least one operation from the current state. Additionally or alternatively, the policy can be trained in such a way that the technical system needs the minimum number of operations to achieve the target state.

[0024] Further, it is proposed that the current state of each of the plurality of domains is characterized at least by the maximum cost that each heuristic method of a predefined set of heuristic methods can return, the minimum cost that each heuristic method of the predefined set of heuristic methods can return, the average cost returned from each heuristic method of the predefined set of heuristic methods, the variance of the costs returned from each heuristic method of the predefined set of heuristic methods, the number of states maintained by each heuristic method of the predefined set of heuristic methods, and the current time step.

[0025] A time step is a time step of a sequence of time steps starting from a first state of the technical system, wherein each time step is assigned to a state of the technical system.

[0026] Advantageously, the state features are domain independent, which inform the policy about characteristics and preferably about the behavior of the planning process, so that the same features can be used for a wide variety of domains. Furthermore, the computational cost of such state features should be low in order to keep the overhead as low as possible, making it suitable for time-critical systems or situations.

[0027] Further, it is proposed that the state further comprises features reflecting context information of the current domain. The domain can be reflected with state features describing e.g. variables, operators or causal graphs, as exemplarily shown by Sievers, S.; Katz, M.; Sohrabi, S.; Samulowitz, H.; and Ferber, P in “Deep learning for cost-optimal planning: Task dependent planner selection” published in 2019. The benefit of this is that the policy can better adapt to different domains and be able to make more precise decisions.

[0028] If the goal is to learn a robust policy that can handle a set of highly heterogeneous domains, context information about the planning domain at hand can be added, such as the problem size or required preprocessing steps, as exemplarily shown by Fawcett, C.; Vallati, M.; Hutter, F.; Hoffmann, J.; Hoos, H.; and Leyton-Brown, K. in “Improved features for runtime prediction of domain-independent planners” (Proc. ICAPS 2014) in 2014.

[0029] Further, it is proposed to subsequently perform steps i) to iv) several times until the current state corresponds to the target state, wherein the selected state with the lowest cost is stored in a list, wherein depending on the list, an operation sequence is determined which generates a sequence of states of the list for reaching the target state. Optionally, a heuristic is determined for all previously expanded states stored in the list.

[0030] This procedure, i.e. the state with the lowest cost is expanded and its subsequent states are added to the list, can be referred to as heuristic search.

[0031] Additionally or alternatively, the policy can be trained in such a way that it minimizes the number of state expansions in the heuristic search.

[0032] The advantage of using the policy for the heuristic search is that the search performance is exponentially improved, because it helps to reduce the search effort and thus improves the performance of the planner. Because the policy selects the heuristic which has the shortest expected planning time, this planning time then increases exponentially, so that just one heuristic selected by the policy is enough to expand the search space to find a path to the target state. Thus, less states are stored in the list, which exponentially improves the performance.

[0033] If more than one target state is defined, it is enough to perform steps i) to iv) until at least one target state is completed.

[0034] Further, it is proposed to control the technical system depending on the operation sequence or to determine a trajectory for controlling the technical system.

[0035] Further, it is proposed to use a list for each heuristic and the most promising state in each list has a corresponding list which is expanded, wherein the successor states are added to all lists and evaluated with the corresponding heuristic of the respective list.

[0036] This has the advantage that the search progress is shared between the heuristics, which leads to a more efficient search procedure.

[0037] Further, it is proposed that the set of heuristics comprises at least one of the following heuristics: a fast forward planning heuristic, or a causal map heuristic, or a context enhanced additive heuristic, or an additive heuristic.

[0038] The following publications describe the heuristic methods: Hoffmann, J. and Nebel, B. in 2001, "The FF planning system: Fast plan generation through heuristic search" (JAIR 14:253-302); and

[0039] Helmert, M. in 2004, "A planning heuristic based on causal graph analysis" (in Proc. ICAPS 2004, 161-170); and

[0040] Helmert, M. and Geffner, H. in 2008, "Unifying the causal graph and additive heuristics" (in Proc. ICAPS 2008, 140-147); and

[0041] Bonet, B. and Geffner, H. in 2001, "Planning as heuristic search" (AIJ 129(1):5-33).

[0042] Further, it is proposed to train the policy via reinforcement learning.

[0043] The advantage of reinforcement learning is that simulations show that the trained policy can recover the optimal policy almost.

[0044] During reinforcement learning, the policy receives state information and, in particular, the cost of all heuristic methods, and learns from it which heuristic method in the set of heuristic methods is potentially the best one for the given domain and state.

[0045] Further, it is proposed that the policy is trained by dynamic algorithm control (DAC).

[0046] Further, it is proposed to utilize a sparse reward function. The sparse reward function ignores aspects such as the quality of the plan, but its aim is to reduce the search effort and, thus, to improve the search performance. BRIEF DESCRIPTION OF DRAWINGS

[0047] Embodiments of the present application will be discussed in more detail below with reference to the following drawings. The drawings show, among others:

[0048] Figure 1 is a control system with a classifier that controls an actuator in its environment;

[0049] Figure 2 is a control system for controlling at least partially autonomous robots;

[0050] Figure 3 is a control system for controlling manufacturing machines;

[0051] Figure 4 is a control system for controlling imaging systems. DETAILED DESCRIPTION

[0052] Planning algorithm with single heuristic method h The general idea of a planning algorithm with a single heuristic method is to start from an initial state and to expand the most promising states based on the heuristic method h until the goal state is found. During the search, relevant states are stored in an open list, which is ordered in ascending order according to the values returned by the heuristic method for the respective states, such that the state with the lowest heuristic value, i.e. the most promising state, is on top. More precisely, in each step, the state with the smallest heuristic value s is expanded, i.e. its successors are generated, and states that have not yet been expanded are added to the open list according to their heuristic values h(s) In the open list, for states with the same heuristic value (h-value), tie-breaking rules can be used according to a first-in-first-out principle.

[0053] In satisfactory planning, multiple heuristic values for the same state can be combined in an arbitrary way. However, it has been shown that combining multiple heuristic values into one value, e.g. by taking the maximum or (weighted) sum, does not lead to a heuristic estimate that provides useful information. This can be explained by the fact that if one or more heuristic methods provide very inaccurate values, the whole expansion process is affected. It is proposed to maintain multiple heuristic methods in a one-greedy best-first search. More precisely, for each heuristic method Maintain separate lists of open heuristics and switch between them at each expansion step, always expanding the most promising state in the currently selected list. The resulting successor state is then added to all open heuristics and evaluated using the corresponding heuristic function. This allows for shared search progress. In particular, a predetermined alternating selection has been shown to be effective, where all heuristics are selected one after another in a loop, ensuring that all heuristics are treated and used equally. This equal use of heuristics can help move the search space toward the target state, even if only one heuristic provides useful information. However, in some cases, it can be inferred that some heuristics currently (i.e., in the current search space) provide more useful information than others, and alternating selection ignores this. Because it is an alternating selection, the choice of heuristic depends only on the current time step, not on the current search dynamics or planner state.

[0054] It can maintain a collection of heuristic methods. H Each heuristic is associated with a separate open list to allow for alternation between such heuristics.

[0055] Will H View as a heuristic search algorithm A Configuration space And each state extension is considered as a time step. t Different dynamic heuristic selection strategies can be classified within the framework of algorithm configuration.

[0056] For example, alternation is an adaptive control strategy because it maps each time step to a specific heuristic (i.e., configuration), independent of the planner's domain or state. Choosing a specific heuristic based on the current domain before solving for the domain is an algorithmic selection strategy that depends only on the domain, not on the current time step or the planner's internal state.

[0057] The inventors discovered that all three components—the domain, the time step, and the planner's state—can be very important and helpful in choosing the heuristic for the next state expansion.

[0058] Therefore, the inventors propose a dynamic algorithm configuration strategy trained via reinforcement learning. This is to reduce the amount of search work and thus improve the planner's performance.

[0059] In a preferred embodiment, a learning strategy is proposed. Dynamic Algorithm Configuration (DAC).

[0060] DAC is a state-of-the-art meta-algorithm framework that allows the tuning of the algorithm's hyperparameters given a description of the algorithm's behavior.

[0061] DAC operates as follows: given a parameterized algorithm A and its configuration space The set of problem domains that this algorithm must solve I In time step Solving domain Algorithm A State description s and reward signals r The reward signal r Evaluation of usage strategies To control the domain On A The reward. The goal is to find a strategy. Given A at time t state of time s Adapt to configuration in the case of This optimizes the reward across domain sets. Note the current time step. t Heyu Can be encoded in the algorithm A In the state description.

[0062] At each time step t The planner will display the current internal state. s ~ and corresponding rewards r Send to policy The controller is based on this strategy Determine which heuristic to use The planner advances to the next internal state based on the decision and its reward.

[0063] For the reward function of the DAC, a reward of -1 is proposed for each expansion step the planning system must take to find a solution. Using this reward function, the configuration policy will learn to choose a heuristic that minimizes the expected number of state expansions until a solution is found. This reward function can be called a sparse reward function and ignores aspects such as the quality of the planning, but its purpose is to reduce the search effort and thus improve search performance. Obviously, other reward functions with, for example, dense rewards can be defined to make learning easier.

[0064] This strategy learns which heuristic will provide the most information at each step of the solution search. It learns through trial and error using RL in simulation. All heuristics can be run in parallel, and the list of heuristics is then updated. Preferably, the RL strategy receives characteristics about the values ​​of all heuristics as its state.

[0065] Preferably, the policy is a neural network. Simulations have shown that a 2-layer network with approximately 75 hidden units and linear decay from 1 to 0 in 5x10^5 steps works best and can learn a policy whose performance is close to the optimal policy.

[0066] Figure 1 One embodiment of an actuator 10 in its environment 20 is shown in Fig. 1. The actuator 10 interacts with a control system 40. The actuator 10 and its environment 20 will be referred to jointly as an actuator system. At preferably evenly spaced distances, sensors 30 sense the state of the actuator system. The sensors 30 can comprise several sensors. Preferably, the sensors 30 are optical sensors that take images of the environment 20. The output signal S of the sensor 30 (or, in case the sensors 30 comprise several sensors, the output signal S for each of the sensors) that encodes the sensed condition is transmitted to the control system 40.

[0067] Thereby, the control system 40 receives a stream of sensor signals S. It then computes a sequence of actuator control commands A depending on the stream of sensor signals S, which it then transmits to the actuator 10.

[0068] The control system 40 receives the stream of sensor signals S of the sensors 30 in an optional receiving unit 50. The receiving unit 50 transforms the sensor signals S into input signals x that describe the state s of the actuator system. Alternatively, in case there is no receiving unit 50, each sensor signal S can be taken directly as an input signal x.

[0069] The input signals x are then passed to a control policy 60, which can for example be given by an artificial neural network.

[0070] The control policy 60 is parameterized by parameters that are stored in and provided by a parameter storage .

[0071] The control policy 60 determines from a set of heuristic methods HThe selection of a heuristic method is described. The heuristic method includes information to assign one or more tags to the input signal x. The heuristic method is transmitted to processor 45, which determines the heuristic method y for returning the next state with the lowest cost. Processor 45 determines the corresponding operation that must be executed by the actuator to reach the next state. This corresponding operation is referred to as the output signal y. An optional conversion unit 80 converts the output signal y into a control command A. The actuator control command A is then transmitted to actuator 10 for corresponding control of actuator 10. Alternatively, the output signal y can be directly used as the control command A.

[0072] Actuator 10 receives actuator control command A, is controlled accordingly, and executes an action corresponding to actuator control command A. Actuator 10 may include control logic that transforms actuator control command A into further control commands, which are then used to control actuator 10.

[0073] In a further embodiment, the control system 40 may include sensor 30. In an even further embodiment, the control system 40 may alternatively or additionally include actuator 10.

[0074] Furthermore, the control system 40 may include a processor 45 (or more processors) and at least one machine-readable storage medium 46 on which instructions are stored, which, when executed, cause the control system 40 to perform a method according to one aspect of the invention.

[0075] Preferably, the present invention can be used to improve the performance of problem-solving algorithms in which a set of heuristics is available. In particular, the present invention can help search algorithms find solutions faster by selecting the best heuristic used in each step. For example, such search algorithms can be applied to find the optimal path for a mobile robot in a path planning problem, or to find the optimal allocation of jobs for available transport robots or production machines in a scheduling problem.

[0076] In a scheduling context, a set of jobs with varying durations. J It requires a collection of machines with various characteristics. M Distributing tasks among different machines. For example, in the semiconductor industry, different machines are used for different parts of the production process: etching, deposition, photolithography, etc. Some machines can complete multiple batches of jobs at once, while others can only handle one job at a time. Scheduling all jobs so that they are all completed in the shortest possible time is a computationally challenging problem, which is usually solved using numerical solvers.

[0077] To make searches more efficient, many different heuristics (or assignment rules) can be used, such as:

[0078] • First In First Out (FIFO) would schedule the first arrived job first,

[0079] • Earliest Due Date (EDD) would preferentially handle jobs with the earliest due date (i.e. the date the customer expects earliest),

[0080] • Shortest Processing Time (SPT) would schedule the job with the shortest processing time first. It is well known that using this heuristic leads to shorter cycle times,

[0081] • Highest Value First (HVO) would schedule the job with the highest value to the customer first,

[0082] • Weighted SPT (WSPT) is a version of SPT that also takes into account the value of the job.

[0083] When these heuristics are combined to find the best solution, a search algorithm can use WSPT to fill the first job in the job queue, EDD to fill the second job, HVO to fill the third job, and so on. This strategy has learned which method is the best heuristic, i.e. which method provides the most information, at each step of scheduling the processing.

[0084] Figure 2 An embodiment is shown in which the control system 40 is used to control an at least partially autonomous robot, for example an at least partially autonomous vehicle 100, in particular for the scheduling situation described above.

[0085] The sensors 30 can comprise one or more video sensors, and / or one or more radar sensors, and / or one or more ultrasonic sensors, and / or one or more LiDAR sensors, and / or one or more positioning sensors (such as e.g. a GPS). Some or all of these sensors are preferably but not necessarily integrated in the vehicle 100.

[0086] Alternatively or additionally, the sensors 30 can comprise an information system for determining a state of the actuator system. One example of such an information system is a weather information system, which determines a current or future state of the weather in the environment 20.

[0087] The actuators 10, which are preferably integrated in the vehicle 100, can be given by brakes, propulsion systems, engines, powertrains or steering devices of the vehicle 100.

[0088] In a further embodiment, the at least partially autonomous robot can be given by another mobile robot (not shown) which can move, for example, by flying, swimming, diving or walking. The mobile robot can be, among other things, an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In all of the above embodiments, the actuator command control A can be determined such that the propulsion units and / or the steering devices and / or the brakes of the mobile robot are controlled such that the mobile robot can avoid a collision with the identified object.

[0089] In a further embodiment, the at least partially autonomous robot can be given by a gardening robot (not shown) which uses a sensor 30, preferably an optical sensor, to determine the state of a plant in the environment 20. The actuator 10 can be a nozzle for spraying a chemical. Depending on the identified species and / or the identified plant state, the actuator control command A can be determined to cause the actuator 10 to spray the appropriate amount of chemical to the plant.

[0090] In a still further embodiment, the at least partially autonomous robot can be given by a household appliance (not shown), such as, for example, a washing machine, a stove, an oven, a microwave or a dishwasher. The sensor 30, for example, an optical sensor, can detect the state of an object to be subjected to the household appliance. For example, in the case of the household appliance being a washing machine, the sensor 30 can detect the state of the laundry inside the washing machine. The actuator control signal A can then be determined depending on the material of the detected laundry.

[0091] Figure 3 An embodiment is shown in Fig. 3, in which the control system 40 is used to control a manufacturing machine 11 of a manufacturing system 200, for example, as part of a production line, for example, a punch, a cutter or a gun drill. The control system 40 controls an actuator 10 which in turn controls the manufacturing machine 11.

[0092] The sensor 30 can be given by an optical sensor which captures characteristics of, for example, a manufactured product 12. The control strategy 60 can determine the state of the manufactured product 12 from these captured characteristics. The actuator 10 which controls the manufacturing machine 11 can then be controlled depending on the determined state of the manufactured product 12 for a subsequent manufacturing step of the manufactured product 12. Alternatively, it can be envisaged that the actuator 10 is controlled depending on the determined state of the manufactured product 12 during the subsequent manufacturing of the manufactured product 12.

[0093] Figure 4An embodiment of a control system 40 for controlling an imaging system 500, for example an MRI apparatus, an X-ray imaging apparatus or an ultrasound imaging apparatus, is shown in Fig. 5. The sensor 30 can for example be an imaging sensor. The machine learning system 60 can then determine a classification of the whole or part of the sensed image. An actuator control signal A can then be selected in dependence on the classification, thereby controlling the display 10a. For example, the machine learning system 60 can interpret a region of the sensed image as being potentially anomalous. In this case, the actuator control signal A can be determined to cause the display 10a to display the imaging and highlight the potentially anomalous region.

Claims

1. A computer-implemented method for planning an operation of a technical system (10) in its environment (20), wherein the technical system being a robot or a transportation system, wherein the operation corresponds to a predefined movement of the robot or the transportation system, The environment is characterized by: a set of different domains ( I The current domain in ) i ), the current state in the state set of each domain ( s A set of possible operations, which can be performed by the technical system in each domain, wherein the domain characterizes the structure, classification, and / or conditions of the technical system's environment. the method comprising the steps, wherein steps i) to iv) are subsequently executed several times until a current state corresponds to a target state, wherein a selected state with a lowest cost is stored in a list, wherein depending on the list, a sequence of operations is determined, which generates a sequence of states of the list for reaching the target state: i) obtaining state information comprising at least a current domain of the environment (D i ), a time step (t t ), and a current state (s s ); ii) determining a cost of a plurality of reachable states from the current state by each heuristic method of a predefined set of heuristic methods, wherein a heuristic method is configured to estimate a cost of reaching the target state from a given state; wherein the set of heuristic methods comprises at least one of the following heuristic methods: a fast forward planning heuristic method, or a causal graph heuristic method, or a context enhanced additive heuristic method, or an additive heuristic method, wherein, for each heuristic method, a list is used and extended with the most promising state of lowest cost of the corresponding list of heuristic methods selected according to a policy π ) the most promising state of lowest cost of the corresponding list of heuristic methods selected according to a policy iii) selecting, depending on the status information, a heuristic method from a predefined set of heuristic methods, according to a policy π ) Among them, strategy ( π It has been trained to select heuristics from a predefined set of heuristics such that the expected number of state expansions is minimized if a path to the target state is planned. iv) selecting from the reachable states the state with the lowest cost determined by the heuristic method selected according to the policy ( π ) and v) determining an operation of the technical system from a set of possible operations that the technical system has to perform to reach the state with the lowest cost determined by the selected heuristic method.

2. The method of claim 1, wherein, The current state of each of the plurality of domains is characterized at least by: a maximum cost that each heuristic method in the predefined set of heuristic methods can return, a minimum cost that each heuristic method in the predefined set of heuristic methods can return, an average cost returned from each heuristic method in the predefined set of heuristic methods, a variance of the costs returned from each heuristic method in the predefined set of heuristic methods, a number of states maintained by each heuristic method in the predefined set of heuristic methods, and a current time step (t) t ).

3. The method of claim 2, wherein, the state further comprises a feature reflecting context information of the current domain.

4. The method according to any of the preceding claims, wherein, the policy is trained via reinforcement learning.

5. The method of claim 4, wherein, the policy is trained by dynamic algorithm control (DAC).

6. The method of claim 4 or 5, wherein, a sparse reward function is utilized.

7. A computer program configured to make a computer execute the method according to any one of claims 1 to 6 and all its steps when the computer program is executed by a processor (45, 145).

8. A machine readable storage medium (46, 146) having stored thereon the computer program according to claim 7.

9. A system configured to execute the method according to any one of claims 1 to 6.

Images