Apparatus and method for training a control policy by means of reinforcement learning

Abstract

A method of training a control policy by means of reinforcement learning is described, which has: performing multiple reinforcement learning training passes, in each pass, for each state of a sequence of states of an agent, selecting an action to be performed starting from an initial state of the pass, for at least some states, selecting the action by specifying a planning horizon, which specifies a number of states; determining, by applying a resolution of answer set programming solver to an answer set programming program, which models relationships between actions and subsequent states reached by these actions, a plurality of sequences of states reachable from the respective state with the specified number of states, selecting a sequence among these sequences that provides a maximum reward, wherein the reward provided by the determined sequence is a sum of rewards obtained upon reaching the states of the sequence; and selecting, as the action for the respective state, an action that can be used to reach the first state of the selected sequence starting from the respective state.

Description

Technical Field

[0001] In general, the various embodiments relate to an apparatus and method for training control policies by means of reinforcement learning. Background Technology

[0002] Control devices for machines like robots can be trained using reinforcement learning (RL) to perform specific tasks, such as those in a production process. Task execution typically involves selecting actions for each state in a sequence of states; it can be viewed as a sequential decision-making problem. These actions are rewarded based on the states reached through the chosen actions, particularly the final state. This reward is determined, for example, by whether the action allows reaching the final state that is rewarding to the action (e.g., for achieving the objective of the task).

[0003] Reinforcement learning enables agents (such as robots) to learn from experience by adjusting their behavior to maximize the rewards they receive over time. Simple trial-and-error-based RL methods exist where the agent does not need knowledge of the control scenario and, given enough time, is guaranteed to converge to the optimal control policy. However, in practice, convergence to the optimal control policy can be very slow, especially in control scenarios where finding rewards is difficult.

[0004] What is desirable are efficient methods that can accelerate the learning process by using prior knowledge about the control scenario, such as by forming models, for example, models of the behavior of the environment. Summary of the Invention

[0005] According to various embodiments, a method for training a control policy by means of reinforcement learning is provided, the method comprising: performing multiple reinforcement learning training traversals, wherein in each reinforcement learning training traversal, for each state of a sequence of agent states, an action to be performed is selected starting from the initial state of the control traversal, wherein for at least some of these states, the corresponding action is selected by specifying a planning horizon, the planning horizon specifying the number of states; determining multiple sequences of states reachable from the corresponding states by applying a response set programming solver to a response set programming procedure, the response set programming procedure modeling the relationship between actions and subsequent states reached through these actions; selecting from the determined sequences the sequence that provides the maximum reward among the determined sequences, wherein the reward provided by the determined sequence is the sum of the rewards obtained when reaching the state of the sequence; and selecting an action that can be used to reach the first state of the selected sequence from the corresponding state as the action for that corresponding state.

[0006] According to another embodiment, a control device is provided, which is configured to control a robotic device by implementing the above-described method or a control strategy trained according to the above-described method.

[0007] Compared to RL methods without a planning component, the above-described method and control device can significantly improve training speed even with a relatively small planning range (e.g., specifying only a small number of states, such as 2 to 10). Therefore, RL methods can be used particularly in practice, for example, in scenarios where the robotic device must learn during runtime (e.g., in real-time adaptation to constantly changing conditions, such as terrain). Due to the limitation of the planning range, a planning component (implemented by a solver programmed from the answer set) is not required to find the final state (i.e., the state at the end of the training traversal), which is difficult or impossible to achieve in some control scenarios. Instead, a planning component with a relatively small planning range (e.g., insufficient to reach the final state at least for the initial state of the training traversal) is repeatedly used (multiple times during the training traversal) until the final state is finally reached.

[0008] This method can be used with any off-policy-RL method, which retains the properties of off-policy-RL methods such as convergence and optimal control policies, but accelerates learning by using prior knowledge (which is introduced as a model through a response set programming procedure) and planning. The (model-based) planning component guides the exploration, but the agent learns from real-world experience (e.g., determined by sensor data). Thus, detailed information about the environment that is not reflected in the model (which exists as a response set programming procedure) is preserved. This allows for the use of simplified or overly optimistic models.

[0009] Various embodiments are described below.

[0010] Example 1 is the method described above for training control strategies using reinforcement learning.

[0011] Example 2 is the method according to Example 1, wherein for a state reached in a reinforcement learning training traversal, it is checked whether the state is the first time it has been reached in the multiple reinforcement learning training traversals, and if the state is the first time it has been reached in the multiple reinforcement learning training traversals, an action is determined by determining the plurality of sequences, selecting the sequence that provides the maximum reward among the determined sequences, and selecting an action that can be used to start from the state to reach the first state of the selected sequence.

[0012] This ensures that prior knowledge contained in the answer set programming is used for each state.

[0013] Example 3 is the method according to Example 2, wherein for a state reached in the multiple reinforcement learning training traversals, the action is selected either according to the control policy trained so far or randomly.

[0014] If a state has already been reached, the prior knowledge has been incorporated into the action selection for that state in one go. By avoiding using the planning component for states that have already been visited, it ensures that the training time is not unnecessarily extended due to the use of the planning component.

[0015] Example 4 is a method according to any one of Examples 1 to 3, wherein for at least some of these states, a corresponding action is selected by specifying a first planning range, the first planning range specifying a first number of states;

[0016] By applying a response set programming solver to a response set programming procedure, multiple sequences of states reachable from a given state are determined. This response set programming procedure models the relationship between actions and subsequent states reached through those actions; and

[0017] If a specified computational budget is secured to determine the action for a given state, then the sequence providing the maximum return among the determined sequences having that first number of states is selected, and the action that can be used to reach the first state of the selected sequence from the given state is selected as the action for that given state; and

[0018] If a specified computational budget has not yet been secured to determine the action for the corresponding state, a second planning scope is specified, which specifies a second number of states, wherein the second number of states is greater than the first number of states; by applying a response set programming solver to a response set programming procedure, multiple sequences of states reachable from the second number of states are determined, the response set programming procedure modeling the relationship between actions and subsequent states reached through these actions; the sequence that provides the maximum reward among the determined sequences of the second number of states is selected; and the action that can be used to reach the first state of the selected sequence from the corresponding state is selected as the action for the corresponding state.

[0019] In this way, the computational cost incurred for planning can be adjusted. In particular, this RL method can be adapted to a given time constraint (e.g., in the case of learning during runtime) by appropriately specifying a computational budget. This computational budget can be, for example, a time budget or a budget for arithmetic operations.

[0020] Example 5 is the method according to any one of Examples 1 to 4, wherein the answer set programming solver supports multiple solutions, and wherein the multiple sequences of states in a reinforcement learning training traversal are determined by means of the answer set programming solver through multiple solutions.

[0021] Using multiple solutions reduces the computational and time costs required for the program's components.

[0022] Example 6 is a control method that controls a robotic device based on a control strategy trained according to any one of Examples 1 to 5.

[0023] Example 7 is a control device configured to implement the method according to any one of Examples 1 to 6.

[0024] Example 8 is a computer program having program instructions that, when implemented by one or more processors, cause the one or more processors to perform the method according to any one of Examples 1 to 6.

[0025] Example 9 is a computer-readable storage medium having stored thereon program instructions that, when implemented by one or more processors, cause the one or more processors to perform the method according to any one of Examples 1 to 6. Attached Figure Description

[0026] Embodiments of the invention are shown in the accompanying drawings and described in more detail below. In the drawings, the same reference numerals generally refer to the same parts in various views. These drawings are not necessarily to scale, but their emphasis is generally on illustrating the principles of the invention.

[0027] Figure 1 The robotic device is shown.

[0028] Figure 2 The interaction between a learning agent and its control environment is illustrated, wherein the agent uses a planning component according to an embodiment of the invention.

[0029] Figure 3 A flowchart illustrating a method for training control policies using reinforcement learning is shown. Detailed Implementation

[0030] Various implementations, particularly the embodiments described below, can be implemented using one or more circuits. In one implementation, "circuit" can be understood as any type of logical implementation entity, which can be hardware, software, firmware, or a combination thereof. Thus, in one implementation, "circuit" can be hardwired logic circuitry or programmable logic circuitry, such as a programmable processor, for example, a microprocessor. "Circuit" can also be software implemented or carried out by a processor, such as any type of computer program. According to an alternative implementation, any other type of implementation of the corresponding functionality can be understood as "circuit," and these corresponding functionalities are described in more detail below.

[0031] Figure 1 Robotic device 100 is shown.

[0032] Robotic device 100 includes a robot 101, such as an industrial robot arm for manipulating or mounting workpieces or one or more other objects 114. Robot 101 has manipulators 102, 103, 104 and a base (e.g., a support) 105 that supports these manipulators 102, 103, 104. The term "manipulator" refers to the movable parts of robot 101, the manipulation of which enables physical interaction with the environment, such as for performing tasks. To control the robot, robotic device 100 includes a (robot) control device 106 configured to interact with the environment according to a control program. The last link 104 of manipulators 102, 103, 104 (farthest from the support 105) is also referred to as an end effector 104 and may include one or more tools, such as a welding torch, a gripper, a painting device, etc.

[0033] Other manipulators 102, 103 (closer to base 105) may form a positioning device such that robot 101, which has an end effector 104 at its end, is provided together with the end effector 104. Robot 101 is a robotic arm that can perform functions similar to a human arm (possibly using tools at its end).

[0034] Robot 101 may include link elements 107, 108, and 109 that connect manipulators 102, 103, and 104 to each other and to a support 105. Link elements 107, 108, and 109 may have one or more links, each of which can realize rotational motion (i.e., rotational movement) and / or translational motion (i.e., displacement) of its associated manipulator relative to each other. The movement of manipulators 102, 103, and 104 can be actuated by actuators controlled by control device 106.

[0035] The term "actuator" can be understood as a component configured to influence a mechanism or process in response to manipulation. The actuator can convert a command (so-called activation) output by control device 106 into mechanical motion. The actuator, such as an electromechanical converter, can be configured to convert electrical energy into mechanical energy in response to the manipulation.

[0036] Robot 101 may have sensors 113 (such as one or more cameras, position sensors, etc.) that are configured to determine the state of the robot and one or more manipulated objects 114 (i.e., the environment), as such state is derived from the manipulation of actuators and the motion produced by the robot.

[0037] In the current example, the control device 106 has a memory 111 and one or more processors 110, the memory storing program code and data, based on which the processors 110 control the robot 101. According to various embodiments, the control device 106 controls the robot 101 according to a control strategy 112 implemented by a control apparatus.

[0038] The approach to generating control policies is reinforcement learning (RL). Reinforcement learning is characterized by trial-and-error search and delayed rewards. Unlike supervised learning, which requires labels for training neural networks, RL uses a trial-and-error mechanism to learn the assignment of states to actions, maximizing the reinforcement signal known as the return. In the case of RL, the goal is to find actions that (ultimately) elicit higher rewards through trial and error. However, finding a good control policy can be very time-consuming.

[0039] According to various implementations, reinforcement learning is accelerated by providing a planning component that includes a model of a controlled system (e.g., a robot and the environment it influences or that influences the robot) in order to recommend promising actions to an agent.

[0040] Here, we can see that the main challenge lies in the need to carefully weigh the additional computing resources required for the plan against the advantages gained from guiding the agent to a more rewarding state.

[0041] In different implementations, the program components (and especially the model) are implemented using answer set programming (ASP). ASP is a declarative problem-solving approach that combines a high-level modeling language for describing the search space of candidate solutions to a given problem with an efficient solver for efficiently computing such solutions.

[0042] Response set programming offers a simple yet powerful modeling language for describing controlled systems in a compact manner. This language includes optimization functions that allow for the optimization of rewards. The search for plans (i.e., sequences of states or corresponding actions) that maximize rewards within a given set of plans can then be expressed using appropriate optimization statements.

[0043] Answer set programming solvers (ASP solvers) can be used to search for optimal solutions within a given range of plans. ASP solvers are particularly suitable when these problems are difficult to solve and involve many constraints. Furthermore, control strategies can be designed to enable multi-shot solving, allowing computation to be reused whenever possible, thereby improving computational speed.

[0044] To describe the implementation of the program component using answer set programming, the terminology of reinforcement learning (RL) will be introduced below.

[0045] In reinforcement learning, the interaction between the agent and the environment (e.g., the robot's interaction with its actions and the environment influencing the state the robot reaches in a given action) is formalized as a Markov decision process (MDP). An MDP has a finite set of states. For each state In other words, Is the agent in this state? A finite set of actions that can be taken. If the agent is in time... In state And carry out the action Then the agent will be at the next time point. Get into the zone And obtain a finite set of numerical rewards. Rewards (For example, a robot receives a reward when it moves an object to a desired location). Agent-environment interaction in state. The initial trajectory has the following form:

[0046]

[0047] The dynamics of this interaction are achieved through functions.

[0048]

[0049] To give, among which It is in state Execution Time to get into the zone And thereby receive a reward. The probability of.

[0050] A control strategy (e.g., a control policy) specifies the appropriate action to take in a given state. The agent's goal is to improve its control strategy based on experience, ensuring that the agent can perform actions in any given state. All of these factors contribute to its expected discounted return.

[0051]

[0052] Maximize, where It is the discount rate, which represents the present value of future rewards.

[0053] Control policies can be improved by learning a value estimation function. This value estimation function is a mapping from a state or state-action pair to the expected reward. This value estimation function (or simply "value function") is updated according to the agent's interaction with the environment. A core problem in reinforcement learning lies in the trade-off between exploring (unexplored) actions and using (exploiting) what has already been learned. When the agent follows a more exploratory control policy... In order to improve its current target control strategy When the target control strategy should ultimately become optimal, the method is called an off-policy method. Control strategy Also known as behavior control strategies.

[0054] An example of an off-policy approach is Q-learning. In the case of Q-learning, the agent learns a value estimation function. This value estimation function maps state-action pairs to expected discounted returns. A goal-oriented control strategy is a type of... The relevant greedy control strategy. If the agent is in a state... Then the agent follows the behavior control strategy. Choose an action The agent observes (e.g., determines based on sensor data) the reward obtained for this. and the state achieved through that reward Then, the agent updates its Q function as follows:

[0055]

[0056] in It is the step size parameter that weights the new experience relative to the old estimate. If all state-action combinations within the bounds (Limes) are studied infinitely frequently according to the behavioral policy, then the target control policy converges to the optimal control policy.

[0057] The drawback of Q-learning is that the Q-value is always updated for only one state-action pair. If a high reward is obtained at some point, it may continue for many iterations (i.e., many updates to the Q-estimation function according to the above formula) until the corresponding update of the Q-estimation function becomes apparent in the initial state. To avoid this, various implementations delay the update until a high reward is observed, and then apply the Q-function update in the reverse order of the observed rewards, so as to efficiently propagate the high estimate (i.e., the high value of the Q-estimation function for some state) back to the initial state.

[0058] Answer set programming is a declarative problem-solving approach rooted in knowledge representation, non-monotonic reasoning, logic programming, and deductive databases. Answer set programming solvers utilize the ideas of SAT (Satisfiability) solvers and SMT (Satisfiability Modular Theories) solvers, but implement non-monotonic semantics that can draw conclusions based on new information. In various implementations, to implement the planning components, the characteristics of problem-solving (e.g., actions and resulting states) are first modeled using the input language of the answer set programming solver. Then, the answer set programming solver is used to compute the answer set of the model, which in turn corresponds to the solution to the initial problem. The capabilities of answer set programming are based on both an expressive yet simple modeling language and a high-performance answer set programming solver.

[0059] Answer set programming is essentially a propositional formalism, and for most answer set programming solvers, variables in the input language are replaced with constant symbols in a preprocessing step called grounding. Besides variables, common features of the input language for answer set programming solvers include integrity constraints, standard negation for expressing lack of information, selection rules and disjunctive reasoning for expressing uncertainty, aggregation, arithmetic, interpreted and uninterpreted function symbols, weak conditions, and optimization specifications.

[0060] A response set programming procedure is a set of rules, where the rules have the following form:

[0061] p,....,q :- r,...,s, not t,..., not u.

[0062] The head of a rule consists of all atoms before the implicating symbol :-, while the body consists of all atoms after the symbol :-.

[0063] The intuitive meaning of this rule is: if all atoms r,...,s can be derived and there is no evidence for any atom t,...,u, then at least one of p,...,q must be true. An interpretation is a set of atoms. An interpretation is the set of responses of a program; if this set of responses satisfies a specific fixed-point condition, then that fixed-point condition ensures that all rules of the program are satisfied in a minimal and consistent manner. A program may have no set of responses, one set of responses, or more than one set of responses.

[0064] Rules with an empty body are called facts. In the case of facts, the implication symbol is usually omitted. Facts are used to express knowledge that is unconditionally true. Rules with an empty head are conditions:

[0065] :- r,...., s, not t,..., not u

[0066] The condition indicates that its body cannot be satisfied by any set of responses. This condition is used to block unwanted solution candidates.

[0067] The following are selection rules:

[0068] { p,...,q} :- r,....,s, not t,..., not u

[0069] The selection rule states that if the main part of the rule is satisfied, then a subset of p,...,q must also be true.

[0070] For illustration, let's assume, as an example, that the robot can push the door at any point in time. The result of pushing the door is that the door opens at the next time step. The corresponding procedure would be, for example:

[0071] { push(T)} :- Time step(T), turn off(T).

[0072] Open (T+1): - Time step (T), Push (T).

[0073] Closed (T+1): - Time step (T), Closed (T), Not open (T+1).

[0074] Open (T+1): - Time step (T), Open (T), Not closed (T+1).

[0075] The first rule states the choice: either push the door if it's closed, or do nothing. The second rule states the effect of pushing the door, i.e., the door subsequently opens. The last two rules are the frame axioms, which state that the state of the door remains unchanged unless there is contrary evidence.

[0076] As an example scenario, consider a unique time step and the door initially closed, that is...

[0077] Time step (1).

[0078] Close (1).

[0079] The program's answer set is

[0080] { Time step (1) Close (1) Push (1) Open (2)}

[0081] and

[0082] { Time step (1) Off (1) Off (2)}

[0083] Each of these answer sets corresponds to a possible world in which the agent either opens a door or does nothing.

[0084] In control scenarios, dynamic functions It is usually not entirely known. However, there is often knowledge about the control scenario that can be utilized. According to various embodiments, this prior knowledge is represented as a set of responses in a programming procedure, thereby implementing the planning component.

[0085] Figure 2 The interaction between the learning agent 201 and the control environment 202 is explained, wherein the agent uses a plan component 208 based on answer set programming with an answer set programming solver 203 and an environment model 204 to select actions.

[0086] Environment Model 204 is a response set programming program. This answer set programming program models the environment (in which the agent interacts). This environment model specifically models the states the agent reaches when performing certain actions. Parameters This specifies the scope of the plan, that is, the maximum number of actions. For a given state... In terms of program The set of answers (with) As a fact, corresponding to the agent according to model 204, from those with the most Each action and its associated rewards The starting trajectory. In one implementation, an optimization criterion for maximizing the reward is described, and the answer set programming solver 203 outputs only one or more answer sets that satisfy the optimization criterion, i.e., the trajectory with the highest reward within the planned range.

[0087] MDP incorporates a probability distribution, while Model 204 models rewards and state transitions deterministically (but optimistically) or nondeterministically. In response set programming, rewards are modeled using either positive or negative integers.

[0088] use To represent the computation results of the answer set programming solver, which will have been set to Model of the scope of the plan With the state of being a fact The symbols are used together as input.

[0089] In one implementation, discounting is not considered because the finite scope of the program ensures that returns cannot tend to infinity. However, it should be noted that discounting can still be used when assessing the capabilities of an agent.

[0090] Below, an example of an environment model 204 matching the above example in which a robot can open a door is described.

[0091] Time step (1..h).

[0092] { push(T)} :- Time step(T), turn off(T).

[0093] Open (T+1): - Time step (T), Push (T).

[0094] Closed (T+1): - Time step (T), Closed (T), Not open (T+1).

[0095] Open (T+1): - Time step (T), Open (T), Not closed (T+1).

[0096] Reward (T, -1): - Push (T-1).

[0097] Reward (T, 10): - Open (T).

[0098] #Maximize { R,T : Reward(T,R)}.

[0099] The first line defines the scope of the plan, in which This is a constant, set by the agent, for example, before the answer set programming solver 203 starts. The reward is modeled in the last three lines: each push action is penalized with a negative reward of -1. However, the agent receives a reward of 10 in every state where the door is open. Intuitively, it is clear that the agent will push the door as quickly as possible to maximize the reward.

[0100] In calling a model Time range 3 and the time during which the door is closed. In other words In the case of a response set programming solver, the response set programming solver provides the response set. equal

[0101] { Time step (1) Time step (2) Time step (3) Close (1) Open (2) Open (3) Open (4) Push (1) Reward (2, -1) Reward (2, 10) Reward (3, 10) Reward (4, 10)}.

[0102] This corresponds to the trajectory where the agent pushes the door open in the first state and then stops taking action.

[0103] Now, the behavior control strategy for agent 201 is as follows. Assume the agent observes the state... (That is, it is observed that it is in a state) If the agent sees this state for the first time (during learning). The agent then uses a response set programming solver to compute the set. And the agent is from the set Choose the first action (that is, the action with a time step of 1) as the action for that state. The action. If the agent has already accessed the state. The solver for the first or answer set of programming problems has no output for that state. The set of available actions The agent then takes action from The agent selects a random action with probability ε and follows the agent's target control strategy with probability (1 - ε). (As if the target control policy were trained at the current training level). Besides the model... In addition to the answer set programming solver 203 used and answer set programming solver parameters such as computation time limits, the planned scope... Both the random exploration rate ε and the random exploration rate are learning parameters.

[0104] If agent 201 takes action at 205, it receives a reward 206 from environment 202 and observes the subsequent state 207 reached. Starting from the initial state, the agent repeats this loop until the corresponding RL training rollout ends, for example, because the final state has been reached or because the maximum number of actions has been reached. RL training typically involves multiple such rollouts until a convergence criterion is met (e.g., when the change in the Q-estimated function for a certain number of updates falls below a threshold).

[0105] Following the strategy described above, at the start of the learning process, a long sequence of similar answer sets and programming solver calls is typically obtained. This occurs if the agent observes a sequence of unknown states.

[0106] ,

[0107] The agent then executes the following sequence of calls to the answer set programming solver:

[0108]

[0109] This means that the answer set programming solver 203 solves closely related problems sequentially. Therefore, to reduce planning costs, one implementation uses multi-shot solving, which some answer set programming solvers support to maintain the solver's state while operating with a constantly changing program. The sequence of calls made by the answer set programming solver can be viewed as an incremental planning problem with a sliding planning range, where the starting point of the trajectory is fixed:

[0110]

[0111] Instead of relying on separate answer set programming solver calls, agent 201 can implement the computation sequence through incremental updates, starting from the first answer set programming solver call. This reduces the computational overhead incurred by planning component 208.

[0112] In summary, according to different implementation methods, a method is provided such as... Figure 3 The methods described in the document.

[0113] Figure 3 Flowchart 300 is shown, which illustrates a method for training control policies using reinforcement learning.

[0114] Multiple reinforcement learning training traversals 301 are executed. In 304, in each reinforcement learning training traversal 301, for each state 302 and 303 of the agent's state sequence, the action to be executed is selected starting from the initial state 302 of the control traversal.

[0115] For at least some of states 302 and 303, the appropriate actions are selected as follows:

[0116] In 305, the plan scope is specified, which specifies the number of states;

[0117] In 306, by applying a response set programming solver to a response set programming procedure, multiple sequences of states reachable from the corresponding states are determined, having a specified number of states. The response set programming procedure models the relationship between actions and subsequent states reached through these actions.

[0118] In 307, from the determined sequences, the sequence that provides the maximum reward among these determined sequences is selected, wherein the reward provided by the determined sequence is the sum of the rewards obtained when reaching the state of that sequence;

[0119] In step 308, an action that can be used to reach the first state of the selected sequence from the corresponding states 302 and 303 is selected as the action for the corresponding state 302 and 303.

[0120] The control strategy corresponds, for example, to the content referred to as the control strategy in the above example.

[0121] In other words, according to various implementations, specific behavior control strategies are defined that provide access to the answer set programming solver when a decision should be made regarding the action to be taken. This method can be used with any off-policy approach to enable the agent to utilize prior knowledge while maintaining the robustness of the (original) off-policy approach. The prerequisites for the various embodiments are only that: states can be represented as relationships between objects such that these states can be used as inputs to the planning components; and the declarative model of the environment is specified in advance in detail using the input language of the answer set programming solver used.

[0122] The underlying idea of ​​various embodiments may be to define a behavioral control strategy that enables the agent to find high rewards in environments where high rewards are scarce. According to various embodiments, the control strategy is a mixture of random exploration, using (exploiting) learned conditions, and planning in the context of using (e.g., advanced) environmental models.

[0123] The learned control strategy is, for example, a control strategy for a robotic device. A "robotic device" can be understood as any physical system (with mechanical parts whose motion is controlled), such as a computer-controlled machine, vehicle, household appliance, power tool, manufacturing machine, personal assistant, or access control system.

[0124] Various implementations can receive and utilize sensor signals from various sensors, such as video, radar, LiDAR, ultrasound, motion, sound, thermal imaging, etc., to obtain sensor data, for example, regarding the system state (robot and one or more objects) and the control scenario. The sensor data can be processed. This may include classifying the sensor data or performing semantic segmentation of the sensor data, for example, to identify the presence of objects (within the environment in which this sensor data was obtained). Implementations can be used to train machine learning systems and autonomously control robots to perform different manipulation tasks in different scenarios. In particular, implementations can be applied to control and monitor the implementation of manipulation tasks, such as those in an assembly line. These implementations can, for example, be seamlessly integrated into conventional GUIs used for process control.

[0125] In one implementation, the method is computer-based.

[0126] Although the invention has been shown and described primarily with reference to specific embodiments, those skilled in the art will understand that numerous changes can be made to its design and details without departing from the spirit and scope of the invention as defined by the following claims. Therefore, the scope of the invention is determined by the appended claims and is intended to cover all changes falling within the literal meaning or equivalent scope of the claims.

Claims

1. A method for training a control policy using reinforcement learning, the method comprising: Multiple reinforcement learning training iterations are performed, wherein in each reinforcement learning training iteration, for each state in the agent's state sequence, an action to be performed is selected starting from the initial state of the control iteration, wherein for at least some of the states, the corresponding action is selected by specifying a plan range, the plan range specifying the number of states. By applying a response set programming solver to a response set programming procedure, multiple sequences of states reachable from corresponding states are determined, having a specified number of states, and the response set programming procedure models the relationship between actions and subsequent states reached through those actions. From the determined sequences, the sequence that provides the maximum reward is selected, wherein the reward provided by the determined sequence is the sum of the rewards obtained when reaching the state of the sequence; and Choose the first action that can be used to reach the selected sequence from the corresponding state, as the action for the corresponding state. For at least some of the states, corresponding actions are selected by specifying a first planning range, wherein the first planning range specifies a first number of states; By applying a response set programming solver to a response set programming procedure, a plurality of sequences of states reachable from the first number of states are determined, and the response set programming procedure models the relationship between actions and subsequent states reached through the actions. Furthermore, if a specified computational budget is raised to determine the action for the corresponding state, the sequence that provides the maximum return in the determined sequence is selected from the determined sequence having the first number of states, and the action that can be used to reach the first state of the selected sequence from the corresponding state is selected as the action for the corresponding state. Furthermore, if the designated computational budget has not yet been secured to determine the action for the corresponding state, then A second plan range is specified, which specifies a second number of states, wherein the second number of states is greater than the first number of states. By applying the answer set programming solver to the answer set programming procedure, multiple sequences of states reachable from the second number of states are determined. The answer set programming procedure models the relationship between actions and subsequent states reached through those actions. From the determined sequence having the second number of states, select the sequence that provides the maximum return among the determined sequences, and Choose an action that can be used to reach the first state of the selected sequence from the corresponding state, as the action for the corresponding state.

2. The method of claim 1, wherein for a state reached in a reinforcement learning training traversal, it is checked whether the state is the first time it has been reached in the multiple reinforcement learning training traversals, and if the state is the first time it has been reached in the multiple reinforcement learning training traversals, the action is determined by determining the plurality of sequences, selecting the sequence that provides the maximum reward among the determined sequences, and selecting an action that can be used to reach a first state from the state to the selected sequence.

3. The method of claim 2, wherein for the states reached in the multiple reinforcement learning training traversals, the action is selected either according to the control policy trained so far or randomly.

4. The method according to any one of claims 1 to 3, wherein the answer set programming solver supports multiple solutions, and wherein the multiple sequences of states in a reinforcement learning training traversal are determined by means of the answer set programming solver through multiple solutions.

5. A control method comprising: controlling a robotic device based on a control strategy trained according to any one of claims 1 to 4.

6. A control device configured to implement the method according to any one of claims 1 to 5.

7. A computer program product comprising a computer program having program instructions that, when implemented by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon program instructions that, when implemented by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 5.

Images