Apparatus and method for controlling a robotic device

Abstract

A device and method for controlling a robotic device are disclosed, wherein the method (400) has: performing a sequence of actions (402) by the robotic device using a robot control model; determining an updated policy (404) using the performed sequence of actions; projecting the updated policy onto a projected policy such that for each state of a plurality of states of the projected policy: a maximization is obtained according to a similarity value of a similarity measure between the projected policy and the updated policy, and the similarity value according to the similarity measure between the projected policy and an initial policy is greater than a predefined threshold (406); adapting the robot control model to implement the projected policy (408); and controlling the robotic device using the adapted robot control model (410).

Description

Technical Field

[0001] Various embodiments generally relate to apparatus and methods for controlling robotic devices. Background Technology

[0002] Robot control models can be used to control robotic devices. For this purpose, the robot control model can be trained using machine learning techniques such as reinforcement learning. Here, the robot control model can select an action to be performed by the robot device based on its current state using a policy for a target. It appears that the policy for a given state among multiple states maps to one of multiple actions. This policy can be updated during the training of the robot control model and / or during inference of the trained robot control model. Here, the similarity between the initial policy and the updated policy may be desired and / or necessary within a predefined region (e.g., a confidence region).

[0003] A Trust Region Policy Optimization (TRPO) method is described in Schulman et al.'s "Trust Region Policy Optimization," ICML, Proceedings of Machine Learning Research, 37, 2015. In this method, policy updates are performed under the condition that the updated policy lies within the trust region. In this paper, this condition is described using a heuristic approximation of the Kullback-Leibler (KL) divergence between the initial and updated policies using the average KL divergence.

[0004] However, in reinforcement learning, the exploration-exploitation compromise (also known as the exploration-exploitation dilemma) must be considered.

[0005] In Abdolmaleki et al.'s "Model-based relative entropy stochastic search," Advances in Neural Information Processing Systems, 2015, it is described that if the entropy of the updated policy is not considered when updating the policy, it may lead to premature policy convergence due to augmentation. When updating the policy within the confidence region, the entropy of the policy can be taken into account as an additional condition.

[0006] Akrour et al.'s "Projections for Approximate Policy Iteration Algorithms," ICLR, 2019, builds upon the TRPO method described by Schulman et al. and the additional conditions of policy entropy described by Abdolmaleki et al., in which the updated conditionally constrained policy is projected onto the unconstrained policy.

[0007] However, the conditions used in TRPO, and therefore the projection of those conditions, are based on the average KL divergence over all states. Therefore, individual states of the projected policy may violate the conditions of the confidence region (e.g., lie outside the confidence region). Thus, it may be necessary to provide a method that can ensure the confidence region for each state when updating the policy.

[0008] Furthermore, the described confidence region policy optimization and policy projection are limited to the average KL divergence. Therefore, it may be helpful and / or necessary to use other mathematical methods to project the policy into the confidence region, such as mathematically more suitable methods (e.g., mathematical methods that require less computational effort, such as mathematical methods that can be solved in closed-form).

[0009] Furthermore, what is advantageous and / or necessary for end-to-end training of robot control models is to provide a method for projecting a policy into a confidence region, by means of which the policy projection is implemented in a neural network as one or more differentiable layers.

[0010] In Amos and Kolter's "OptNet: Differentiable Optimization as a Layer in Neural Networks", 34 th A method was described at the International Conference on Machine Learning, 2017, which integrates optimization problems as differentiable hierarchies into neural networks (OptNet). Summary of the Invention

[0011] The method (first example) and the device (eighteenth example) according to the invention can train a robot control model such that a confidence region (e.g., a corresponding confidence region) is ensured for each state of the robot control model when updating the strategy of the robot control model.

[0012] Therefore, the devices and methods used to control robotic devices can train the robot control model more effectively (e.g., faster, with higher accuracy, for example, with an improved exploration and utilization ratio).

[0013] The robot control model can be a machine learning-based model. For example, the robot control model may incorporate a reinforcement learning algorithm. According to various embodiments, at least a portion of the robot control model can be implemented using a neural network.

[0014] Robotic devices can be any type of computer-controlled device, such as robots (e.g., manufacturing robots, maintenance robots, home robots, medical robots, etc.), vehicles (e.g., autonomous vehicles), home appliances, production machines, personal assistants, access control systems, etc.

[0015] By projecting an updated policy onto the robot control model in a manner that ensures a confidence region for each state of the robot control model, it is possible to control (e.g., improve, e.g., optimize) exploration in reinforcement learning by leveraging compromises.

[0016] Determining an updated policy using the sequence of executed actions can involve: determining a reward for each action in the sequence by applying a reward function to the resulting state; and determining the updated policy using the initial policy and the determined rewards to maximize the desired reward. The features described in this paragraph, combined with the first example, form a second example.

[0017] Projecting the updated policy onto the projected policy can have the following characteristics: The updated policy is projected onto the projected policy such that, for each of the multiple states of the projected policy: the similarity value measured by the similarity metric between the projected policy and the updated policy is maximized; the similarity value measured by the similarity metric between the projected policy and the initial policy is greater than a predefined threshold; and the entropy of the projected policy is greater than or equal to a predefined entropy threshold. The features described in this paragraph, combined with the first or second example, form a third example.

[0018] This can be manifested as the condition that the entropy of each state in a multi-state policy is greater than or equal to a predefined entropy threshold, causing changes in not only the covariance but also the expected value of the multivariate normal distribution of the projected policy when updating the policy.

[0019] The initial policy may have an initial multivariate normal distribution with multiple actions. The updated policy may have an updated multivariate normal distribution with multiple actions. The projected policy may have a projected multivariate normal distribution with multiple actions. Projecting the updated policy onto the projected policy may involve: projecting the updated policy onto the projected policy such that for each of the multiple states of the projected policy: the similarity value measured by the similarity metric between the projected multivariate normal distribution and the updated multivariate normal distribution is maximized, and the similarity value measured by the similarity metric between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than a predefined threshold. The features described in this paragraph, combined with one or more of the first to third examples, form a fourth example.

[0020] Projecting the updated policy onto the projected policy can have the following characteristics: The updated policy is projected onto the projected policy such that, for each of the multiple states of the projected policy: the similarity value measured by the similarity metric between the projected multivariate normal distribution and the updated multivariate normal distribution is maximized, and the similarity value measured by the similarity metric between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than a predefined threshold; and the entropy of the projected multivariate normal distribution is greater than or equal to a predefined entropy threshold. The features described in this paragraph, combined with the third and fourth examples, form the fifth example.

[0021] The updated policy is projected onto the projected policy such that, for each of the multiple states of the projected policy: the similarity value measured by the similarity metric between the projected multivariate normal distribution and the updated multivariate normal distribution is maximized, and the similarity value measured by the similarity metric between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than a predefined threshold, which can be achieved by: determining the projected multivariate normal distribution using the initial multivariate normal distribution, the updated multivariate normal distribution, and the predefined threshold with the aid of Mahalanobis distance and the Frobenius norm. The features described in this paragraph, combined with the fourth or fifth example, form the sixth example.

[0022] The updated policy is projected onto the projected policy such that, for each of the multiple states of the projected policy: the similarity value measured by the similarity metric between the projected multivariate normal distribution and the updated multivariate normal distribution is maximized, and the similarity value measured by the similarity metric between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than a predefined threshold, which can be achieved by: determining the projected multivariate normal distribution using the initial multivariate normal distribution, the updated multivariate normal distribution, and the predefined threshold via Wasserstein distance. The features described in this paragraph, combined with the fourth or fifth example, form the seventh example.

[0023] Using the Mahalanobis distance and Frobenius norm according to the sixth example, or the Wasserstein distance according to the seventh example, has the following effect: the projection of the updated policy can be determined in a mathematically closed form. For example, the projected policy determined in this way can be integrated into one layer (or multiple layers) of a neural network.

[0024] The updated policy is projected onto the projected policy such that, for each of the multiple states of the projected policy: the similarity value measured by the similarity metric between the projected multivariate normal distribution and the updated multivariate normal distribution is maximized, and the similarity value measured by the similarity metric between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than a predefined threshold, which can be achieved by: using the initial multivariate normal distribution, the updated multivariate normal distribution, and the predefined threshold to determine the projected multivariate normal distribution with the aid of a numerical optimizer. The features described in this paragraph, combined with the fourth or fifth example, form the eighth example.

[0025] The numerical optimizer can use Kullback-Leibler divergence to determine the projected multivariate normal distribution. The features described in this section, combined with the eighth example, form the ninth example.

[0026] The projected multivariate normal distribution can be determined using the Lagrange multiplier method. The features described in this section, combined with one or more of the examples from the sixth to the ninth, form the tenth example.

[0027] The robot control model can be a neural network. The features described in this paragraph, combined with one or more of the first to tenth examples, form the eleventh example.

[0028] Projecting the updated policy onto the projected policy can be implemented as one or more layers in a neural network (e.g., differentiable layers). The features described in this section, combined with the eleventh example, form the twelfth example.

[0029] Integrating the policy into a neural network as one or more differentiable layers by projecting the policy onto the state's own confidence region has the following effect: the neural network can be trained end-to-end using policy projection, wherein the condition of the confidence region is ensured for each state during training.

[0030] Adapting the robot control model to achieve the projected policy can be achieved using a gradient method. The features described in this paragraph, combined with one or more of the first through twelfth examples, form the thirteenth example.

[0031] Using an adapted robot control model to control the robotic device can have the following characteristics: the robotic device performs one or more actions using the adapted robot control model; and the policy is updated using the performed one or more actions by means of regression. The features described in this paragraph, combined with one or more of the first through thirteenth examples, form the fourteenth example.

[0032] Using an adapted robot control model to control the robotic device can have the following characteristics: the robotic device performs one or more actions using the adapted robot control model; and the executed one or more actions are used to update the policy such that the difference between the expected reward and the similarity value of the similarity metric between the projected policy and the updated policy is maximized. The features described in this paragraph, combined with one or more of the first through thirteenth examples, form the fifteenth example.

[0033] A method for controlling a robotic device may include: the robotic device executing a sequence of actions using a robot control model, wherein executing each action in the sequence of actions comprises: determining an action for a current state among a plurality of states of the robotic device using an initial policy via the robot control model; the robotic device executing the determined action; and determining a state of the robotic device resulting from the executed action; determining an updated policy using the executed sequence of actions; determining a projected policy such that the difference between the expected reward for the projected policy and a similarity metric between each of the plurality of states according to the projected policy and the updated policy is maximized; adapting the robot control model to implement the projected policy; and controlling the robotic device using the adapted robot control model. A method having the features described in this paragraph forms a sixteenth example.

[0034] A method for controlling a robotic device may include: the robotic device executing a sequence of actions using a robot control model, wherein executing each action in the sequence of actions comprises: determining an action for a current state among a plurality of states of the robotic device using an initial policy via the robot control model; the robotic device executing the determined action; and determining a state of the robotic device resulting from the executed action; determining an updated policy using the executed sequence of actions; determining a projected policy such that the difference between the expected reward for the projected policy and a similarity metric between each of the plurality of states according to the projected policy and the updated policy is maximized; and controlling the robotic device using the projected policy via the robot control model. A method having the features described in this paragraph forms the seventeenth example.

[0035] A computer program product may store program instructions that, when executed, perform a method according to one or more of the first to seventeenth examples. A computer program product having the features described in this paragraph forms the nineteenth example.

[0036] The non-volatile storage medium can store program instructions that, when executed, perform one or more methods from the first to the seventeenth examples. A non-volatile storage medium having the features described in this paragraph forms the twentieth example.

[0037] The non-volatile storage medium can store program instructions that, when executed, perform one or more methods from the first to the seventeenth examples. A non-volatile storage medium having the features described in this paragraph forms the twenty-first example. Attached Figure Description

[0038] Embodiments of the present invention are shown in the accompanying drawings and are explained in more detail in the following description.

[0039] in:

[0040] Figure 1 Exemplary robot equipment arrangements according to various embodiments are shown;

[0041] Figure 2 An illustrative flowchart for determining a strategy according to various implementation methods is shown;

[0042] Figure 3 A graph is shown illustrating how a numerical optimizer is used to determine the projected strategy according to various implementations;

[0043] Figure 4 Methods for controlling robotic devices according to various embodiments are illustrated;

[0044] Figure 5 Methods for controlling robotic devices according to various embodiments are shown. Detailed Implementation

[0045] In one implementation, "computer" can be understood as any type of logical implementation entity, which can be hardware, software, firmware, or a combination thereof. Thus, in one implementation, "computer" can be hardwired logic circuitry or programmable logic circuitry, such as a programmable processor, such as a microprocessor (e.g., CISC (Big Data Set Processor) or RISC (Reduced Instruction Set Processor)). "Computer" can have one or more processors. "Computer" can also be software implemented or executed by a processor, such as any type of computer program, such as a computer program using virtual machine code such as Java. Consistent with an alternative implementation, any other type of implementation of the corresponding functionality described in more detail below can be understood as "computer".

[0046] Robotic devices can be controlled using reinforcement learning-based robot control models. To ensure an improved compromise between exploration and exploitation (e.g., optimality) when updating the policy of the robot control model, it may be necessary to update the policy within a confidence region. Various embodiments relate to apparatuses and methods for controlling robotic devices, which are capable of training a robot control model such that the updated policy for each state of the robotic device lies within the said confidence region. This can be manifested in considering and ensuring the confidence region for each state of the robotic device when updating the policy of the robot control model.

[0047] Figure 1 A robot device arrangement 100 is shown. The robot device arrangement 100 may include a robot device 101. For illustration, in... Figure 1 The robot device 101 shown and described exemplary below represents an exemplary robot device and may, for example, be an industrial robot in the form of a robotic arm for moving, installing, or processing workpieces. Note: The robot device can be any type of computer-controlled device, such as a robot (e.g., a manufacturing robot, a repair robot, a home robot, a medical robot, etc.), a vehicle (e.g., an autonomous vehicle), a home appliance, a production machine, a personal assistant, an access control system, etc.

[0048] Robotic device 101 has robotic limbs 102, 103, 104 and a base (or typically a support) 105 supporting the robotic limbs 102, 103, 104. The term "robotic limb" refers to the movable part of robotic device 101, the actuation of which enables physical interaction with the environment, for example, to perform a task, such as performing an action.

[0049] For control purposes, the robot equipment arrangement 100 includes a control device 106 configured to interact with the environment according to a control program. The last element 104 of the robot limbs 102, 103, 104 (viewed from the base 105) is also called an end effector 104 and may contain one or more tools, such as a welding torch, a clamping tool, a painting device, etc.

[0050] Other robotic limbs 102, 103 (closer to the base 105) can form a positioning device, such that the robotic arm (or articulated arm) is positioned together with the end effector 104, wherein the end effector 104 is located at the end of the robotic arm. The robotic arm is a type of mechanical arm that can perform functions similar to a human arm (possibly with tools at the end of the robotic arm).

[0051] Robotic device 101 may include connecting elements 107, 108, and 109 that connect robot limbs 102, 103, and 104 to each other and to a base 105. Connecting elements 107, 108, and 109 may have one or more joints, each joint providing rotational and / or translational motion (i.e., displacement) relative to its associated robot limb. Movement of robot limbs 102, 103, and 104 can be initiated by means of an adjustment mechanism controlled by control device 106.

[0052] The term "regulating mechanism" can be understood as a component adapted to influence a mechanical device in response to being driven. The regulating mechanism can convert commands (so-called activation) output by the control device 106 into mechanical motion. Regulating mechanisms, such as those of electromechanical transducers, can be configured to convert electrical energy into mechanical energy in response to their manipulation.

[0053] The term "control device" (also known as "control apparatus") can be understood as any type of logical implementation unit that may include, for example, circuitry and / or a processor capable of executing software, firmware, or a combination thereof stored in a storage medium, and capable of issuing instructions, such as to the regulating mechanism in this example. The control device may be configured, for example, via program code (e.g., software) to control the operation of a system, which in this example is a robot.

[0054] In this example, control device 106 includes a computer 110 and a memory 111 for storing code and data, based on which the computer 110 controls the robot device 101. According to various embodiments, control device 106 controls the robot device 101 based on a robot control model 112 stored in memory 111.

[0055] According to various embodiments, the robot device arrangement 100 may have one or more sensors 113. The one or more sensors 113 may be configured to provide sensor data characterizing the state of the robot device. For example, the one or more sensors 113 may include imaging sensors such as cameras (e.g., standard cameras, digital cameras, infrared cameras, stereo cameras, etc.), radar sensors, LiDAR sensors, position sensors, velocity sensors, ultrasonic sensors, acceleration sensors, pressure sensors, etc.

[0056] Robot device 101 can be in multiple states. According to various implementations, the robot device 101 can be in one of multiple states at any given time. The corresponding state among the multiple states can be determined using sensor data provided by one or more sensors 113.

[0057] Robotic device 101 can be configured to perform multiple actions. For example, actions among multiple actions can be predefined in the program code of control device 106. One or more of the multiple actions may have mechanical movements, for example, one or more robotic limbs 102, 103, 104. One or more of the multiple actions may have actions, for example, actions of an end effector (e.g., grasping, releasing, etc.). According to various embodiments, in the current state of robotic device 101... The next action to be executed This can lead to one of the multiple states of robot device 101.

[0058] The robot control model 112 can be a reinforcement learning-based model. For example, the robot control model 112 can implement a reinforcement learning algorithm.

[0059] The robot control model 112 can be configured to determine one action from multiple actions for one of a set of states. For example, the robot control model 112 can output one action from multiple actions in response to an input from one of the states. This can be represented as the robot control model 112 mapping from one state to one action from multiple actions. The states in the multiple states can form a state space. The actions in the multiple actions can form an action space. This can be represented as the robot control model 112 mapping from the state space to the action space.

[0060] According to various implementation methods, the robot control model 112 may have a policy. For example, robot control model 112 can follow a policy at any point in time. A corresponding policy can be assigned to a goal and / or task. For example, the corresponding policy can be a policy for achieving a goal or completing a task. According to various implementations, the policy can output one of a plurality of actions in response to input from one of a plurality of states. This can be represented as the policy used by robot control model 112 mapping from a state space to an action space.

[0061] Multiple actions can be assigned to each of the multiple states, with corresponding probability distributions (e.g., normal distributions). Depending on the implementation, the policy may have or may be a multivariate normal distribution (also known as a multidimensional normal distribution and / or a multivariate Gaussian distribution). A multivariate normal distribution can be defined by an expectation vector and a covariance matrix. The expectation vector of the policy's multivariate normal distribution can have the expectation value of each of the multiple states. The covariance matrix of the policy's multivariate normal distribution (also referred to herein as covariance) can depend on multiple states (e.g., it is a function of the multiple states).

[0062] According to various implementations, the control device 106 can be configured to control the robot device 101, such that the robot device 101 performs and / or implements actions determined by the robot control model 112 using the current strategy for the current state of the robot device 101.

[0063] The control device 106 can be configured to determine a reward for the state of the robot device 101 caused by the performed action. According to various embodiments, the control device 106 can use a reward function to determine the reward resulting in the state. For example, an algorithm for executing the reward function can be stored in memory 111. For example, the robot control model 112 can be configured to execute the reward function. For example, the reward determined for the resulting state can be assigned to an action performed in conjunction with the initial state of the robot device 101.

[0064] According to various implementations, the robot device 101 can use the robot control model 112 to execute a sequence of actions. The control device 106 can be configured to use an initial strategy. Each action in the action sequence is determined. The control device 106 can be configured to determine a corresponding reward for each action performed in the sequence of actions.

[0065] Control device 106 (e.g., computer 110 of control device 106) can be configured to determine the updated policy using the sequence of actions performed. Control device 106 can be configured to determine the updated policy. This increases (e.g., maximizes) the expected reward given in equation (1):

[0066] (1),

[0067] in The states traversed using the aforementioned strategy to achieve the goal or complete the task. and actions The trajectory of , where γ is the discount factor, and where and .

[0068] Strategy The parameters of model 112 can be controlled by the robot. Definition, for example, can be obtained through right Parameterization.

[0069] According to various implementation methods, the updated strategy can be determined using equation (2). :

[0070] (2),

[0071] in It is the initial strategy (such as the strategy used previously), and

[0072] in It is a payoff function. For example, it can be obtained through... To determine the payoff function, where It is the action value function, and It is the value function.

[0073] According to various implementation methods, the updated strategy can be determined using random sampling based on importance. .

[0074] According to various implementation methods, the updated strategy Can obey the initial strategy One or more conditions (e.g., boundary conditions, or secondary conditions). For example, a condition could be an updated strategy. Regarding the initial strategy Within a trust region (e.g., robot control model 112 can implement reinforcement learning based on a trust region). For example, the condition can be each of multiple states. According to the updated strategy Compared with the initial strategy The similarity metric between them is greater than a predefined threshold. For example, if the updated strategy... Compared with the initial strategy The distance d between them is less than or equal to a predefined threshold. Then each of the multiple states Based on the strategy to be used and the initial strategy The similarity value measured between them may be greater than a predefined threshold. Depending on the implementation, the corresponding predefined threshold can be... Assigned to each of the multiple states. For example, the condition (subject to) with respect to equation (2) is that each of the multiple states... According to the updated strategy and initial strategy The similarity value measured between them is greater than a predefined threshold. This condition is described by equation (3):

[0075] (3).

[0076] This can be represented as restricting the updated policy for each point (e.g., the state) in the state space. It can also be represented as ensuring that updating the policy within the confidence region results in the policy approaching the optimal policy with a moderately large step size (e.g., convergence with a moderately large step size). Furthermore, it can be represented as limiting the degree of variation of the policy used.

[0077] According to various implementation methods, the condition can be each of multiple states. Updated strategy entropy Greater than or equal to a predefined entropy threshold For example, the conditions of equation (2) can be described by equation (4):

[0078] (4).

[0079] Using the updated strategy based on equation (3) and optionally further based on equation (4). The relevant conditions enable the exploration (also known as exploration) and exploitation (also known as exploitation) of reinforcement learning in the robot control model 112.

[0080] According to various implementation methods, equation (2) using the conditions of equations (3) and (4) can be defined by means of equation (5):

[0081] (5).

[0082] refer to Figure 2 The robot control model 112 can use an initial strategy. Each action in the action sequence is determined (in 202). As described herein, control device 106 can determine (in 204) the updated strategy according to equation (2). The updated strategy It can be subject to the conditions defined in equation (3) and optionally also to the conditions defined in equation (4). This can be expressed as the updated strategy... It can be a restricted updated strategy Updated strategy Each of the multiple states can be restricted (e.g., subject to conditions according to equation (3)). This can be expressed as the updated policy. It is a state-specific, restricted, updated strategy For example, each of the multiple states can have its own predefined threshold. This makes the predefined threshold It can be a predefined threshold vector.

[0083] Control device 106 can be configured to determine (in 206) the projected strategy. Control device 106 can be configured to update the policy. Projection to post-projection strategy Above. Control device 106 can be configured to update the policy. Projection to post-projection strategy Above, this makes the strategy after projection... For each of the multiple states, increase (e.g., maximize) according to the projected policy. and the updated strategy The similarity value is a measure of the similarity between the two. Control device 106 can be configured to update the policy. Projection to post-projection strategy Above, this makes the strategy after projection... For each of the multiple states, increase (e.g., maximize) according to the projected policy. and the updated strategy The similarity value between them is measured, and the similarity is calculated for the projected strategy. For each of the multiple states, according to the projected policy and initial strategy The similarity value measured between them is greater than a predefined threshold. Control device 106 can be configured to update the policy. Projection to post-projection strategy Above, this makes the strategy after projection... For each of the multiple states, increase (e.g., maximize) according to the projected policy. and the updated strategy The similarity value between them is measured, and the similarity is calculated for the projected strategy. For each of the multiple states, according to the projected policy and initial strategy The similarity value measured between them is greater than a predefined threshold. And for the projected strategy Each of the multiple states, each of the multiple states Updated strategy entropy Greater than or equal to a predefined entropy threshold .

[0084] According to various implementation methods, the corresponding strategy can be described using the assigned multivariate normal distribution. For example, the initial strategy... An initial multivariate normal distribution can have multiple actions. The initial multivariate normal distribution can be described as: ,in It is the initial expected value vector, and It is the initial covariance of the initial multivariate normal distribution. For example, the updated strategy The updated multivariate normal distribution can have multiple actions. The updated multivariate normal distribution can be described as follows: ,in It is the updated expected value vector of the updated multivariate normal distribution, and It is the updated covariance of the updated multivariate normal distribution. The initial expected value vector, initial covariance, updated expected value vector, and / or updated covariance can be functions of multiple states. For example, the projected policy. A projected multivariate normal distribution can have multiple actions. The projected multivariate normal distribution can be described as: ,in It is the expected value vector of the projected multivariate normal distribution, and It is the projected covariance of the projected multivariate normal distribution. The projected expected value vector can depend on the initial expected value vector, the updated expected value vector, the initial covariance, the updated covariance, a predefined threshold, and / or multiple states. The projected covariance can depend on the initial expected value vector, the updated expected value vector, the initial covariance, the updated covariance, a predefined threshold, multiple states, and / or a predefined entropy threshold.

[0085] Control device 106 can be configured to update the policy Projection to post-projection strategy Above, this makes the strategy after projection... For each of the multiple states, increase (e.g., maximize) the similarity value based on the similarity measure between the projected multivariate normal distribution and the updated multivariate normal distribution. Control device 106 can be configured to update the policy... Projection to post-projection strategy Above, this makes the strategy after projection... For each of the multiple states, increment (e.g., maximize) the similarity value based on the similarity metric between the projected multivariate normal distribution and the updated multivariate normal distribution, and for the projected policy... For each of the multiple states, the similarity value measured by the similarity measure between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than a predefined threshold. Control device 106 can be configured to update the policy. Projection to post-projection strategy Above, this makes the strategy after projection... For each of the multiple states, increase (e.g., maximize) the similarity value based on the similarity metric between the projected multivariate normal distribution and the updated multivariate normal distribution, for the projected policy. For each of the multiple states, the similarity value measured by the similarity measure between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than a predefined threshold. And for each state in the multiple states of the projected multivariate normal distribution, each state in the multiple states The updated entropy of the multivariate normal distribution Greater than or equal to a predefined entropy threshold .

[0086] As described in this paper, the distance d can be used to describe the similarity value based on the similarity metric. According to various implementations, based on the projected strategy... The projected multivariate normal distribution and the updated strategy The similarity values ​​of the updated multivariate normal distributions can have a projected strategy. The distance d between the projected multivariate normal distribution and the updated multivariate normal distribution. According to various implementations, based on the projected strategy... The projected multivariate normal distribution and the initial strategy The similarity measure between initial multivariate normal distributions can have a projected strategy. The projected multivariate normal distribution and the initial strategy The distance d between the initial multivariate normal distributions. According to various implementations, the strategy for determining the projection can be described according to equations (6) to (8). Optimization issues:

[0087]

[0088] According to various implementation methods, the updated strategy can be Projection to post-projection strategy Above, so that the projected policy is an unrestricted projected policy. This can be expressed as determining the projected strategy by satisfying these conditions (e.g., see equation (6)). This manifests as the updated strategy. Projection to post-projection strategy Above, such that the projected multivariate normal distribution is as close as possible to the updated multivariate normal distribution (e.g., the distance between the projected multivariate normal distribution and the updated multivariate normal distribution is minimized) and the projected multivariate normal distribution (and therefore the projected strategy) These conditions must be met.

[0089] The projected multivariate normal distribution can be obtained using the projected expected value vector. and projected covariance To describe. The updated strategy Projection to post-projection strategy The vector above can have a definite expected value after projection. and projected covariance .

[0090] This can be represented as projecting the updated multivariate normal distribution onto the projected multivariate normal distribution under one or more conditions described in this paper. Alternatively, it can be represented as projecting the updated multivariate normal distribution onto the projected multivariate normal distribution such that for each of the multiple states, one or more conditions described in this paper are satisfied.

[0091] The following describes the method used to determine the projected expected value vector. and projected covariance Three exemplary projection methods:

[0092] (I) The first projection method using Mahalanobis distance and Frobenius norm;

[0093] (II) A second projection method using Wasserstein distance; and

[0094] (III) The third projection method using a numerical optimizer.

[0095] (I) First projection method

[0096] According to various implementation methods, the expected value vector after projection based on equation (9) can be used. and projected covariance The equation (6) describes the optimization problem using Mahalanobis distance and Frobenius norm:

[0097] (9).

[0098] According to various implementation methods, the expected value vector and covariance can be independent of each other. For example, the expected value vector and covariance can be observed independently. For example, for a predefined threshold of the expected value vector according to equation (10). and the predefined threshold of covariance according to equation (11) We can consider the conditions of equation (7):

[0099]

[0100] This can be represented by observing similarity values ​​based on a similarity measure of the expected value vector and similarity values ​​based on a similarity measure of the covariance. According to various implementations, the optimization problem described by equations (9) to (11) can be solved using the Lagrange multiplier method. For example, the Lagrange function of equation (12) can be used... To describe the Lagrange duality of equations (9) to (11):

[0101] (12),

[0102] in and It is a Lagrange multiplier.

[0103] By solving equation (12), we obtain the expected value vector projected according to equation (13) and the covariance projected according to equation (14):

[0104] (13)

[0105] (14),

[0106] This can be determined according to equation (15). And determined according to equation (16) .

[0107] (15)

[0108] (16).

[0109] (II) Second projection method

[0110] According to various implementation methods, the expected value vector after projection based on equation (17) can be used. and projected covariance The equation (6) describes the optimization problem using the Wasserstein distance (e.g., scaled Wasserstein distance):

[0111] (17),

[0112] Where tr is the trace of the matrix.

[0113] The Wasserstein distance is the Euclidean distance between two normal distributions, which includes the expectations of both distributions. Multiplied by the initial covariance. And scaling the Wasserstein distance results in the Mahalanobis distance (for this, see equation (17) for example).

[0114] As described in this paper, the expected value vector and covariance can be observed independently of each other. For example, for a predefined threshold of the expected value vector according to equation (18). and the predefined threshold of covariance according to equation (19) We can consider the conditions of equation (7):

[0115]

[0116] in It is the identity matrix (also known as the information matrix).

[0117] According to various implementations, the Lagrange multiplier method can be used to solve the optimization problem described by equations (17) to (19). Regarding the projected expected value vector... The solution is given by referring to equations (13) and (15).

[0118] According to various implementation methods, an optimization problem concerning the roots of the projected covariance can be solved. For example, it can be solved using the Lagrangian function of equation (20). To describe the Lagrange duality of equations (17) and (19).

[0119] (20)

[0120] By solving equation (20), the projected covariance according to equation (21) is obtained:

[0121] (twenty one)

[0122] This can be determined according to equation (22). .

[0123] (twenty two)

[0124] Therefore, the first and second projection methods can be solved in a closed form (the multivariate normal distribution after projection can be determined in a closed form).

[0125] (III) Third projection method

[0126] According to various implementation methods, the optimization problem based on equations (6) to (8) can be solved by means of a numerical optimizer.

[0127] Figure 3 The diagram illustrates how the numerical optimizer 302 is used to determine the projected strategy according to various implementations. For example, control device 106 (e.g., computer 110 of control device 106) can be configured to implement numerical optimizer 302.

[0128] Regular expression parameters can be used (Also known as natural parameters) and cumulant generation functions To describe the multivariate normal distribution.

[0129] Numerical optimizer 302 can be configured to target regularization parameters. and cumulant generation function Solving the optimization problem according to equations (6) to (8) is done by numerical optimizer 302 targeting the regularization parameter. and cumulant generation function Determine the first optimal Lagrange multiplier Second optimization Lagrange multiplier For example, the numerical optimizer 302 can use KL divergence to determine the first optimization Lagrange multiplier. Second optimization Lagrange multiplier .

[0130] According to various embodiments, the control device 106 can be configured to... For the updated covariance 304 Determines the updated cumulative generation function 306. For example, for the updated expected value vector 308 and the updated cumulant generation function 306, Control device 106 can be based on Determine the regular expression parameters to update 310.

[0131] Numerical optimizer 302 can be configured to use an updated cumulant generation function. 306 and updated regular expression parameters 310 determines the first optimal Lagrange multiplier. Second optimization Lagrange multiplier The numerical optimizer 302 can be configured to use an updated cumulant generation function. 306. Updated regularization parameters; 310. First Lagrange multiplier. 312 and the second Lagrange multiplier 314 determines the first optimal Lagrange multiplier. 316 and the second optimized Lagrange multiplier 318. For example, the first Lagrange multiplier. 312 and / or the second Lagrange multiplier 314 can be predefined (e.g., set). For example, the numerical optimizer 302 can be set to determine the first Lagrange multiplier. 312 and / or the second Lagrange multiplier 314.

[0132] Initial covariance can be used (For example, based on) To determine the initial cumulant generation function An initial cumulant generation function and an initial expected value vector can be used (e.g., based on...). To determine the initial regularization parameters .

[0133] The regularization parameters after projection can be determined according to equation (23). 320:

[0134] (twenty three).

[0135] Projected Cumulative Generation Function 322 can be determined according to equation (24):

[0136] (twenty four).

[0137] The projected cumulant generation function can be used 322 (e.g., based on) To determine the projected covariance 324. Projected regularization parameters can be used. 320 and the projected cumulative generation function 322 (e.g., based on) To determine the expected value vector after projection .

[0138] Tip: Updated strategy Alternatively, a strategy can be employed to project onto the projected target using methods different from the three exemplary projection methods described in this paper. superior.

[0139] According to various implementations, the projected strategy is an optimization strategy, and by means of the projected strategy, the state of the robot device 101 and the optimal sequence (e.g., order) of actions to be performed and / or implemented by the robot device 101 can be determined and executed by the robot device 101.

[0140] According to various implementations, the robot control model 112 may have either a neural network or a pre-defined strategy. Projection to post-projection strategy This can be implemented as one or more layers in a neural network. For example, the updated policy... Projection to post-projection strategy The above can be implemented as one or more differentiable layers in a neural network. According to various implementations, the one or more layers can be configured to, if the updated policy is not satisfied... If one of the conditions is met, then the projection described in this article will be performed.

[0141] According to various embodiments, the control device 106 can be configured to adapt (e.g., fit) the robot control model 112 to implement the projected strategy. .

[0142] According to various embodiments, the robot control model 112 may have a neural network, and the adaptation of the robot control model 112 may be the training of the neural network. For example, a gradient method (e.g., a policy gradient method) may be used to train the neural network. According to various embodiments, a projected policy may be used. and initial strategy To determine one or more gradients. This can be represented by using a projected strategy. Adapting a neural network involves iteratively training the network. Depending on the implementation, multiple iterations can be performed. For example, the method described herein for adapting a robot control model 112 can be performed multiple times.

[0143] For example, the neural network of robot control model 112 can be adapted (e.g., trained) using one or more determined gradients by means of gradient methods.

[0144] The first and second projection methods can be solved in a closed-form solution. Here, one or more gradients can be directly determined. For the third projection method, one or more gradients can be determined using the OptNet method described by Amos and Kolter. Here, the layers of the neural network can solve for the following Lagrangian duality:

[0145]

[0146] According to various implementation methods, one or more gradients can be determined (e.g., calculated) by deriving the corresponding Karush-Kuhn-Tucker (KKT) conditions.

[0147] The fixed KKT conditions can be described, for example, by using equation (25):

[0148] (25),

[0149] in It is the first KKT multiplier, and It is the second KKT multiplier.

[0150] The complementary slackness of KKT can be described, for example, by means of equation (26):

[0151] (26).

[0152] According to various implementations, one or more gradients can be determined by deriving the Karush-Kuhn-Tucker (KKT) conditions. According to various implementations, one or more layers of the neural network can be configured such that, if the updated policy described herein is not satisfied… If one of the conditions is met (e.g., according to the conditions of equation (3), or according to the conditions of equation (4), then projection is performed. For example, one or more gradients can be determined for the following conditions:

[0153] - Satisfies the conditions according to equation (3) and equation (4). This can be expressed as the updated policy being used in the gradient method in this case. For example, it is not necessary to use one or more layers of a neural network to project the policy and / or compute the gradient.

[0154] - It satisfies the condition according to equation (3) but does not satisfy the condition according to equation (4).

[0155] - It does not satisfy the condition according to equation (3), but satisfies the condition according to equation (4).

[0156] - The conditions according to equation (3) and equation (4) are not satisfied.

[0157] According to various implementations, if at least one of these conditions is not met, one or more layers of the neural network may project the policy as described herein. According to various implementations, if at least one of these conditions is not met and if a third projection method is used, one or more layers of the neural network may determine one or more gradients.

[0158] This can be manifested as one of the three projection methods being implemented as one or more differentiable layers in a neural network, enabling the neural network to be trained end-to-end, ensuring (e.g., satisfying) one or more conditions (e.g., conditions of the confidence region) for each of the multiple states during training.

[0159] According to various implementations, the control device 106 can be configured to control the robot device 101 using an adapted robot control model 112.

[0160] Control device 106 may be configured to determine the current state of robot device 101. Control device 106 may be configured to determine the action to be performed for the current state using a projected strategy via an adapted robot control model 112. The action to be performed may be, for example, the action with the highest probability among multiple actions (e.g., the action with the expected value assigned to the current state), described by a projected multivariate normal distribution. Control device 106 may be configured to control robot device 101 corresponding to the action to be performed, causing robot device 101 to perform and / or implement the action. According to various embodiments, robot device 101 may use the adapted robot control model 112 to perform one or more actions.

[0161] According to various embodiments, the control device 106 can update the policy using one or more actions performed. Here, as described herein, the updated policy can be determined and the projected policy can be determined using the updated policy. According to various embodiments, for example when the inference robot controls the model 112 (e.g., a neural network), an optimization problem according to equation (27) can be solved.

[0162] (27)

[0163] Here, one or more actions performed can be adapted to the robot control model 112 by means of regression (e.g., having one or more regression steps).

[0164] According to various implementations, the projected policy can be determined based on the objective function given in equation (28). Here, the projected policy can be determined as the difference between increasing (e.g., maximizing) the expected reward (see equation (2)) and the similarity value measured by the similarity metric between the projected policy and the updated policy. For example, this can be measured by the distance between the projected policy and the updated policy. This determines the similarity value based on the similarity metric between the projected policy and the updated policy. For example, the distance between the projected policy and the updated policy can be determined using the three projection methods described in this paper.

[0165] (28).

[0166] Figure 4 A method 400 for controlling a robotic device according to various embodiments is shown.

[0167] Method 400 may involve a robotic device using a robot control model to execute a sequence of actions (in 402). Executing each action in the sequence of actions may involve: determining an action for the current state of the robotic device among multiple states using an initial policy via the robot control model; executing the determined action by the robotic device; and determining the state of the robotic device resulting from the executed action. According to various implementations, the robot control model may be a reinforcement learning-based model (e.g., a reinforcement learning-based neural network).

[0168] Method 400 may have the ability to determine the updated policy using the sequence of actions performed (in 404).

[0169] Method 400 may involve projecting the updated policy onto the projected policy (in 406). The updated policy may be projected onto the projected policy such that, for each of the multiple states of the projected policy, a similarity value measured by the similarity between the projected policy and the updated policy is increased (e.g., maximized), and for each of the multiple states of the projected policy, the similarity value measured by the similarity between the projected policy and the initial policy is greater than a predefined threshold. The updated policy may be projected onto the projected policy such that, for each of the multiple states of the projected policy, a similarity value measured by the similarity between the projected policy and the updated policy is increased (e.g., maximized), and for each of the multiple states of the projected policy, the similarity value measured by the similarity between the projected policy and the initial policy is greater than a predefined threshold, and for each of the multiple states of the projected policy, the entropy of the projected policy is greater than or equal to a predefined entropy threshold.

[0170] Method 400 may have an adapted robot control model to implement the projected strategy (in 408).

[0171] Method 400 may have the ability to control the robotic device using an adapted robot control model (in 410).

[0172] Figure 5 A method 500 for controlling a robotic device according to various embodiments is shown.

[0173] Method 500 may involve a robotic device using a robot control model to execute a sequence of actions (in 502). Executing each action in the sequence of actions may involve: determining an action for the current state of the robotic device among multiple states using an initial policy via the robot control model; executing the determined action by the robotic device; and determining the state of the robotic device resulting from the executed action. According to various implementations, the robot control model may be a reinforcement learning-based model (e.g., a reinforcement learning-based neural network).

[0174] Method 500 may have the ability to determine the updated policy using the sequence of actions performed (in 504).

[0175] Method 500 may have a determined projected policy such that it increases (e.g., maximizes) the expected reward for the projected policy and the difference between the similarity values ​​of the similarity measure between each of the multiple states of the projected policy and the updated policy (in 506).

[0176] Method 500 may have the ability to control a robotic device using a projected strategy with the aid of a robot control model (in 508).

[0177] According to various implementations, method 500 may have an adapted robot control model to implement the projected strategy and use the adapted robot control model to control the robot device.

Claims

1. Method for controlling a robotic device, having: • performing a sequence of actions by the robotic device using a robot control model, wherein performing each action in the sequence of actions has: o determining an action for a current state of a plurality of states of the robotic device by means of the robot control model using an initial policy, o performing the determined action by the robotic device, and o determining a state of the robotic device resulting from the performed action; • determining an updated policy using the performed sequence of actions; • projecting the updated policy onto a projected policy such that for each state of a plurality of states of the projected policy: o a similarity value according to a similarity measure between the projected policy and the updated policy is maximized, and o a similarity value according to a similarity measure between the projected policy and the initial policy is greater than a predefined threshold value; • adapting the robot control model to implement the projected policy; and • controlling the robotic device using the adapted robot control model.

2. The method according to claim 1, wherein projecting the updated policy onto a projected policy has: • projecting the updated policy onto the projected policy such that for each state of a plurality of states of the projected policy: o a similarity value according to a similarity measure between the projected policy and the updated policy is maximized, o a similarity value according to a similarity measure between the projected policy and the initial policy is greater than the predefined threshold value, and o an entropy of the projected policy is greater than or equal to a predefined entropy threshold.

3. The method according to any one of claims 1 or 2, wherein the initial policy has an initial multivariate normal distribution of a plurality of actions; wherein the updated policy has an updated multivariate normal distribution of a plurality of actions; wherein the projected policy has a projected multivariate normal distribution of a plurality of actions; wherein projecting the updated policy onto a projected policy has: • projecting the updated policy onto the projected policy such that for each state of a plurality of states of the projected policy: o a similarity value according to a similarity measure between the projected multivariate normal distribution and the updated multivariate normal distribution is maximized, and o a similarity value according to a similarity measure between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than the predefined threshold value.

4. The method according to claim 3, wherein projecting the updated policy onto a projected policy has: • projecting the updated policy onto the projected policy such that for each state of a plurality of states of the projected policy: o a similarity value according to a similarity measure between the projected multivariate normal distribution and the updated multivariate normal distribution is maximized, and o a similarity value according to a similarity measure between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than the predefined threshold value; and o an entropy of the projected multivariate normal distribution is greater than or equal to the predefined entropy threshold.

5. The method according to claim 3, wherein the updated policy is projected onto the projected policy such that for each state of a plurality of states of the projected policy, a similarity value according to a similarity measure between the projected multivariate normal distribution and the updated multivariate normal distribution is maximized and according to a similarity value of a similarity measure between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than the predefined threshold, there is: • determining the projected multivariate normal distribution by means of Mahalanobis distance and Frobenius norm using the initial multivariate normal distribution, the updated multivariate normal distribution and the predefined threshold.

6. The method according to claim 3, wherein the updated policy is projected onto the projected policy such that for each state of a plurality of states of the projected policy, a similarity value according to a similarity measure between the projected multivariate normal distribution and the updated multivariate normal distribution is maximized and according to a similarity value of a similarity measure between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than the predefined threshold, there is: • determining the projected multivariate normal distribution by means of Wasserstein distance using the initial multivariate normal distribution, the updated multivariate normal distribution and the predefined threshold.

7. The method according to claim 3, wherein the updated policy is projected onto the projected policy such that for each state of a plurality of states of the projected policy, a similarity value according to a similarity measure between the projected multivariate normal distribution and the updated multivariate normal distribution is maximized and according to a similarity value of a similarity measure between the projected multivariate normal distribution and the initial multivariate normal distribution is greater than the predefined threshold, there is: • determining the projected multivariate normal distribution by means of a numerical optimizer using the initial multivariate normal distribution, the updated multivariate normal distribution and the predefined threshold.

8. The method according to claim 7, wherein the numerical optimizer determines the projected multivariate normal distribution using Kullback-Leibler divergence.

9. The method according to any one of claims 5 to 8, wherein determining the projected multivariate normal distribution has a Lagrange multiplier method.

10. The method according to claim 1 or 2, wherein the robot control model is a neural network; and wherein projecting the updated policy onto the projected policy is implemented as one or more layers in the neural network.

11. The method according to claim 1 or 2, wherein adapting the robot control model to implement the projected policy has adapting the robot control model by means of a gradient method.

12. The method according to claim 1 or 2, wherein controlling the robotic device using the adapted robot control model has: • performing one or more actions by the robotic device using the adapted robot control model; • updating the policy using the performed one or more actions by means of regression.

13. The method according to claim 1 or 2, wherein controlling the robotic device using the adapted robot control model has: • performing one or more actions by the robotic device using the adapted robot control model; • updating the policy using the performed one or more actions such that a difference between an expected reward and a similarity value of a similarity measure between the projected policy and the updated policy is maximized.

14. A device arranged to perform the method according to any one of claims 1 to 13.

15. Non-transitory storage medium having stored program instructions which, if executed, perform the method according to any one of claims 1 to 13.

Images