Apparatus and method for improving robotic policy learning

Abstract

Apparatus and methods to improve robotic policy learning. A computer-implemented method for learning a policy, comprising the steps of: recording at least episodes of an agent interacting with an environment it follows a policy for, and adding the recorded episodes to a training data set. Optimizing a transition dynamics model based on the training data, such that the transition dynamics model predicts a next state of the environment depending on a state and an action contained in the training data. Optimizing policy parameters based on the training data and the transition dynamics model by optimizing a return. The method is characterized in that the transition dynamics model comprises a first model characterizing a global model and a second model characterizing a correction model configured to correct an output of the first model.

Description

[0001] The present invention relates to a method for learning strategies through an improved environment model, a computer program and a machine-readable storage medium, and a system for implementing the method. Existing technology

[0002] In reinforcement learning, the goal is to control real, unknown stochastic systems, characterized by transition dynamics. p : Equation (1): Its encoding is the probability distribution of the next state. .

[0003] Note that deterministic systems are explicitly allowed as a special case of stochastic systems. The system is governed by a stochastic (or deterministic) strategy. Control, making π θ Depends on time step t and the corresponding state s t , and by θ Parameterization.

[0004] The goal of reinforcement learning is to find the optimal learning method within a given time frame. H Internal expected return r Maximize policy parameters θ Allows for an unlimited range, that is, The resulting optimization problem is: Equation (2): in Indicate the so-called discount factor, and It's a phased return.

[0005] In fact, the exact dynamics p It is unknown. One solution would be to work with real systems. p Data is collected during the interaction and used to optimize the policy. Another way to do this is through model-based reinforcement learning, where an approximate dynamic model is learned based on trajectory data from equation (1). In other words, the current strategy π θ Apply this to the system and record the state transition in line 4 of the following pseudocode: Pseudocode for reinforcement learning based on a general model: .

[0006] The key challenge is that learning models inevitably only approximate the real world, making... Therefore, the expectation in equation (2) lies on different state distributions, which may lead to slow convergence and even in the real system of equation (2). p The suboptimal solution above is the problem that this invention will solve.

[0007] There are two main ways to mitigate model error. One is to improve the model training in the pseudocode in line 5, while an alternative approach focuses on policy optimization in line 6.

[0008] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine's "When to trust your model: Model-based policy optimization" (in Advances in Neural Information Processing Systems, volume 32, pages 12519–12530. Curran Associates, Inc., 2019) proposes a method to optimize a model based on its performance in the field of Neural Information Processing Systems. D env Starting with independently and identically distributed states, the model simulates only a short range. This reduces the magnitude of the error because the simulation error is combined with the length of this range. However, it does not address the fundamental problem of model error from rollout and makes the rollout range a basic (and challenging to tune) hyperparameter.

[0009] Andreas Doerr, Christian Daniel, Duy Nguyen-Tuong, Alonso Marco, Stefan Schaal, Toussaint Marc and Sebastian Trimpe, “Optimizing long-termpredictions for model-based policy search” (volume 78 of Proceedings ofMachine Learning Research, pages 227–238. PMLR, 13–15 Nov 2017. URL Instead, the focus is on learning approximate models to achieve low error in long-term prediction tasks.

[0010] Iterative learning control (DA Bristow, M. Tharayil and AG Alleyne, “A survey of iterative learning control”, IEEE Control Systems Magazine, 26(3):96–114, 2006) describes a class of methods focusing on specific sub-cases of model-based RL, where the policy π Both the learning model and the learning model are fixed, except for the time-dependent error term, which can be computed in closed form based on the linearization of the learning model and the reward function with the state and action as quadratic.

[0011] Advantages of the present invention Propose using data D env Learn or improve (potentially time-related) models from data collected through interactions with the environment. Finally, based on this learning model, the policy parameters are optimized by approximating equation (3) (e.g., in line 6 of the pseudocode): Equation (3): .

[0012] To this end, a specific form of model learning is proposed, which can be appended to other model learning methods to improve learning performance. In particular, while existing methods learn real dynamics based solely on environmental data... p global model g However, an additional learning (on-policy) correction term is proposed. Thus, the resulting model is approximately given by equation (4): Equation (4): It has a global model that can be state-dependent. g Same strategy correction model d t .

[0013] If the global model and / or correction terms d t If it is probabilistic, then the summation operation is interpreted as the sum of random variables encoded by the distribution.

[0014] Although global model g The dynamics are approximated globally (including off-policy, or in other words, for all data collected during the RL process), but same-policy correction ensures low-error same-policy (or in other words, using data already collected with a policy similar to the current policy).

[0015] If from the expected transition kinetics proposed in equation (4) D env The replay action sequence in the plot can obtain accurate plot data from the real environment. Therefore, the error of the same strategy model in equation (3) is reduced, which leads to improved stability and data efficiency when optimizing the strategy.

[0016] In a special case of deterministic transition dynamics and policy, the correction term leads to zero error in the same policy model. Summary of the Invention

[0017] In a first aspect, the present invention relates to a computer-implemented method for learning a strategy for an agent, particularly a robot that is at least partially autonomous. Depending on the current state, the strategy can be configured to output actions or a distribution of actions. If the actions proposed by the strategy are followed, the objective that the strategy has preferably optimized for through reinforcement learning will be achieved.

[0018] This method uses an initialization strategy π θ and transformation dynamics model Initially, if the agent performs an action ( a Then the transformation dynamics model Depends on the current state ( s t ) and the action ( a Both predict the next state in the environment and / or agent. s t+1 The distribution above the next state. Therefore, it can be said that the transition dynamics model approximates the probabilistic or deterministic transitions and outputs the next state. Note that the environment can also be a system, where the agent is, for example, an actor in the system. The transition dynamics model takes the current state and action (e.g., the action determined by the policy) as input and returns the next state or a distribution above the next state.

[0019] Then repeat the following steps until the termination condition is met: By following the strategy π θ And the recorded episodes are added to the training data. D env A set is used to record at least the interaction scenarios between the agent and its environment. A scenario can be the agent's trajectory.

[0020] Recorded interactions can include pairs of states and applied actions, or applied actions and the next state reached thereafter. A plot then includes all pairs from the first action to the last. The steps of recording interactions can be performed in a real-world environment. Alternatively, this step is implemented through environmental simulation. States are typically sensed by sensors, where sensors sense the agent's state and / or the agent's environmental state, and then use this as input to the policy. In the case of environmental simulation, the states are simulated.

[0021] The plot can be collected synchronously or asynchronously, with the same or different strategies, and has a complete or partial trajectory.

[0022] After adding plots to the training data, follow up with plots based on the training data. D env For the transformation dynamics model ( The optimization of the transformation dynamics model () makes the transformation dynamics model () (This depends on what is included in the training data) D env The system uses the states and actions recorded in the data to predict the next state of the environment. Interestingly, all recorded episodes can be reused independently of the process trained on the policy. This property makes this step highly efficient in terms of sampling.

[0023] Transformation dynamics model ( The characteristic of ) is that it includes a first model representing the global model ( g ) and the correction model also known as the same strategy correction model ( d t ), the correction model ( d t The correction model is configured to correct the output of the first model. In other words, the transformation dynamics model is a superposition of the first and second models. In the following text, the correction model is also referred to as the correction term. Transformation Dynamics Model It is configured to predict the next state that the environment and / or agent will be in when performing a predefined action on the current state. Therefore, it predicts the state at the next time step, where the time step can be related to the point in time when the action is performed or the state is sensed. The time step can also be a discrete value, representing the number of actions performed since a predefined starting point. A correction model is configured to correct the error of a first model relative to the true state of the environment after the action is performed. For example, the first model predicts the state of the environment based on the current state and, for example, an action determined by a policy. The correction model then corrects the predicted state of the first model so that, after the agent performs the action on the current state, the predicted state is as similar as possible to the true state of the environment. In other words, the correction model corrects the output of the first model to obtain a predicted state that is as close as possible to the state the environment will present. Therefore, the correction model corrects the first model to obtain a more accurate state regarding the environment, particularly environmental dynamics.

[0024] Note that the transformation dynamics model is learned to approximate the dynamics of the environment, and preferably the dynamics of the agent. This allows the model to be based on the dynamics of the environment. D env The plot data should be used to predict the state as accurately as possible.

[0025] Preferably, the first model depends on the current state and action of the environment, and outputs a predicted state of the environment if the input action is performed. The correction term depends on time or time step and / or the current state. Alternatively, the correction term is an extracted correction value determined by the difference between the recorded state of the training data and the predicted state of the first model. The correction term outputs the correction for the first model. The first model is parameterized, where its parameters are optimized in the step of optimizing the transition dynamics model. In contrast, the output of the correction model can be directly added to the discrete correction of the first model, or the correction term can also be parameterized and optimized on the difference between the recorded state of the training data and the predicted state of the first model. A special case of the correction term is a non-parametric function, instead consisting of a time-discrete error term.

[0026] After the previous step is completed, the following steps are taken by following the (current) strategy. π θ Optimize rewards beyond at least one plot point. r Therefore, based on training data D env and transformation dynamics model Optimization strategy π θ parameter θ The transformation dynamics model This is used to predict the next state when the policy is applied. In other words, the policy is not applied to the environment, but instead interacts with the transition dynamics model. This optimization step can also be called optimizing the policy through model-based reinforcement learning. The optimization involves updating the policy parameters, particularly depending on the reward.

[0027] Given a learning transformation dynamics model Any policy optimization method can be used. Examples include, among many others, soft actor-critic ("Soft actor-critic: Off-policy maximum entropy deepreinforcement learning with a stochastic actor," Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine, 2018), or stochastic value gradients ("Learning continuous control policies by stochastic value gradients," Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa, in "Advances in Neural Information Processing Systems," volume 28, pages 2944–2952, Curran Associates, Inc., 2015. URL) ).

[0028] The strategy can be a neural network, or a deterministic function on any other parameterized distribution or action.

[0029] The reward indicates the quality of the state and / or action taken. Typically, this reward signal is designed to manipulate the behavior of the learning algorithm. Generally, the reward signal assigns large values ​​to desirable states / actions and small (or negative) values ​​to states / actions the agent should avoid. Possible reward signals include, but are not limited to: a negative tracking error of a reference state signal, an indicator function for task-specific success or a smoothed variant thereof (e.g., an exponential function), a negative quadratic cost term (similar to methods from optimal control), etc. If the learning algorithm should strive for multiple objectives simultaneously, it is also possible to construct another reward signal as a weighted sum of some other reward signals.

[0030] If the termination condition is not met, the steps just explained will be repeated, starting with recording the episode. The termination condition can be a convergence criterion chosen relative to policy optimization. Note that if the transition dynamics model is accurate enough, the recording episode step can be skipped.

[0031] In other words, the method proposed in the first aspect extends the transition dynamics model by adding state and / or time-related terms to reduce model predictions for the same policy, i.e., actions chosen by the current control policy. This has the advantage that the transition dynamics model becomes more accurate without additional interaction with the environment. Because the accuracy of the transition dynamics model is improved only for the parts related to the learned policy, particularly for those achieved by following the policy, the method automatically focuses only on improving the relevant parts of the global model.

[0032] A method for optimizing the transformation dynamics model was proposed. After optimizing the first model on the training data, a correction term is selected to make the model more suitable for different training data. D env The above strategy π θ The selected action results in a low transition dynamics model error. A calibration model is applied to the current policy to provide the most accurate transition dynamics model for each convergent state. Preferably, the calibration model is optimized such that the error of the global model is minimized for the current policy, which is the policy currently being optimized based on the newly added training data.

[0033] In addition, the selection of correction terms is proposed. , so that if D env Choosing actions as precisely as in the middle leads to anticipated changes. The sequence will become equal to the training data. That is, the selection function... d t To minimize a certain measure of the difference between the left and right sides of the following equation. Equation (5): .

[0034] Furthermore, a method is proposed to minimize the correction model. d t The output is measured against the following differences to select the correction model. d t Recording status of training data With the transformation dynamics model The first item g The difference between the expected and predicted states.

[0035] In addition, a correction model was proposed. d t With the first item g Joint optimization. Preferably, in order to optimize strategy parameters... θ It can be done through strategy π θ The model is reweighted and corrected based on the probability (or logarithmic probability) of the next action. d t This is to ensure that the model fits the same strategy.

[0036] Joint optimization implies a change in the dynamic model. Both terms are updated within the same optimization step. This has the advantage that the error of the first model does not need to be explicitly determined, nor does it require optimizing the correction term for said error. This speeds up the method and further improves the accuracy of the transformation dynamics model.

[0037] In addition, a correction term was proposed. d t It is state- and / or time-related. This time can be the elapsed time since the start of the scenario. During training, the elapsed time since the start of the scenario is a measurable time from the point in time when the first state of the environment has been sensed until the current point in time when the current state is sensed. The same definition can also be applied when using a strategy obtained according to the first aspect of the invention. Alternatively, time can characterize the number of actions performed since the first action of the scenario.

[0038] In addition, a project was proposed. d t It can only be time-related; in this case, we can have deterministic correction terms. Given the data and the first model, this deterministic correction term is fully specified. If the environment is deterministic, then when taking... Actions in At that time, the time-related term leads to the error of the zero-error model.

[0039] Furthermore, a correction term is proposed as a probability function, which is optimized through approximate inference. When exact learning and inference are computationally intractable, the approximate inference method trades computation time for accuracy, thus making it possible to learn realistic models from big data.

[0040] The determined actions of the strategy can be used to provide control signals for controlling the agent, including all the steps of the methods described above for controlling the robot, and further including the step of determining the actuator control signal based on the output signal. Preferably, the actuator controls at least partially autonomous robots and / or manufacturing machines and / or access control systems.

[0041] Note that a policy can be learned to control the dynamics and / or stability of the agent. This policy can receive sensor values ​​characterizing the state of the agent and / or the environment as input. The policy is trained to follow an optimal trajectory by maximizing stage rewards. The policy outputs a value characterizing the control value, ensuring the agent follows the optimal trajectory.

[0042] Embodiments of the invention will be discussed in more detail with reference to the following figures. The figures illustrate: Figure 1 Flowchart of the method; Figure 2 A control system having a classifier that controls actuators in its environment; Figure 3 A control system for controlling at least partially autonomous robots; Figure 4 Control systems for manufacturing machines; Figure 5 Control system for an automated personal assistant; Figure 6 A control system that controls access control systems; Figure 7 The control system of the control and monitoring system; Figure 8 The control system for the imaging system. Detailed Implementation

[0043] In model-based reinforcement learning, the real system is learned. p Approximate models of probabilistic or deterministic transitions In other words, the learning model takes the current state and control action as input and returns a distribution of the next state or above the next state. Therefore, this model can be used in conjunction with existing policy optimization methods. To optimize the strategy. The proposed same-strategy correction adopts a method of... A fixed or learned approximation is used, and it is extended using state and / or time-related terms to reduce model predictions for the same policy, i.e., under the current control policy. π Under the selected action.

[0044] Figure 1 The diagram shows a flowchart of an embodiment of a method for learning strategies for controlling a robot.

[0045] This method uses an initialization (S1) strategy ( π θ ) and transformation dynamics model ( ) begin if the agent depends on the current state ( s t ) and actions ( a Both perform the actions described. a ), then the transformation dynamics model ( Predict the next state of the environment and / or agent. s t+1 ).

[0046] This continues in a loop until the termination condition is met: The first step of the loop is to record at least (S2) the agent and the policy it follows ( π θ The interactive scenarios within the environment. Subsequently, the recorded scenarios are added (S3) to the training data ( D env A set of ).

[0047] Next, step optimization (S4) is performed. Here, based on the training data ( D env )Optimize the transformation dynamics model ( ), making the transformation dynamics model ( ) depends on the data contained in the training data ( D env The state and actions in the environment are used to predict the next state of the environment.

[0048] Model global part g Any existing method can be used to learn this information, including approximate inference, methods for minimizing prediction errors, and methods for optimizing long-term prediction errors. Correction term. d t It is an optimization relative to Equation 5 above.

[0049] After completing step S4, follow step S5. This step includes proceeding by following a strategy ( π θ And optimize the reward for at least one episode, based on the training data (D env ) and transformation dynamics model ( Optimization strategy π θ )parameter( θ Given the transformation dynamics model of step S4 ( Any policy optimization method can be used. Examples include, among many others, soft participant-commenter, stochastic value gradient, approximate policy optimization, and maximum a posteriori policy optimization.

[0050] If step S5 has been completed and the termination condition is not met, the loop can be repeated.

[0051] If the loop has terminated, the resulting optimized strategy can be used to compute control signals for a physical system, such as a computer-controlled machine, robot, vehicle, household appliance, power tool, manufacturing machine, or access control system. It does this by learning a strategy for controlling the physical system and then operating the physical system accordingly. Generally, the strategy obtained as described above can interact with any kind of system. Therefore, its applications are very broad. Several applications are described below as examples.

[0052] Figure 2 The diagram illustrates one embodiment of actuator 10 within its environment 20. Actuator 10 interacts with control system 40. Actuator 10 and its environment 20 will be collectively referred to as the actuator system. Sensors 30 sense the state of the actuator system at preferably uniformly spaced distances. Sensors 30 may include several sensors. The output signal S of sensor 30, encoding the sensed state (or, in the case of multiple sensors, the output signal S of each sensor), is transmitted to control system 40. Possible sensors include, but are not limited to, gyroscopes, accelerometers, force sensors, cameras, radar, lidar, angle encoders, etc. Note that sensors typically do not directly measure the state of the system, but rather observe the result of the state; for example, a camera detects an image rather than directly detecting the relative position of the car to other traffic participants. However, it is possible to filter the state from high-dimensional observations such as images or lidar measurements.

[0053] Therefore, the control system 40 receives the flow of sensor signals S. It then calculates a series of actuator control commands A based on the flow of sensor signals S, which are then transmitted to the actuator 10.

[0054] The control system 40 receives the sensor signal S stream from the sensor 30 in an optional receiving unit 50. The receiving unit 50 transforms the sensor signal S into a state s. Alternatively, in the absence of the receiving unit 50, each sensor signal S can be directly taken as the input signal s.

[0055] The input signal s is then passed to policy 60, which can be provided, for example, by an artificial neural network.

[0056] Strategy 60 is determined by parameters Parameterization, the parameter Stored in parameter storage device St 1. and provided by it.

[0057] Strategy 60 determines the output signal y from the input signal s. The output signal y can be an action a. The output signal y is transmitted to an optional conversion unit 80, which converts the output signal y into a control command A. The actuator control command A is then transmitted to the actuator 10 for corresponding control of the actuator 10. Alternatively, the output signal y can be directly taken as the control command A.

[0058] Actuator 10 receives actuator control command A, is controlled accordingly, and performs actions corresponding to actuator control command A. Actuator 10 may include control logic that transforms actuator control command A into another control command, which is then used to control actuator 10.

[0059] In another embodiment, the control system 40 may include sensor 30. In even further embodiments, the control system 40 may alternatively or additionally include actuator 10.

[0060] In one embodiment, strategy 60 can be designed to control signals of a physical system, such as a computer-controlled machine, robot, vehicle, household appliance, power tool, manufacturing machine, or access control system. It does this by learning a strategy for controlling the physical system and then operating the physical system accordingly.

[0061] In yet another embodiment, it is conceivable that the control system 40 controls the display 10a instead of the actuator 10.

[0062] Furthermore, the control system 40 may include a processor 45 (or more processors) and at least one machine-readable storage medium 46 thereon storing instructions which, when executed, cause the control system 40 to perform a method according to one aspect of the invention.

[0063] Figure 3One embodiment is shown, wherein the control system 40 is used to control a robot that is at least partially autonomous, such as a vehicle 100 that is at least partially autonomous.

[0064] Sensor 30 may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors and / or one or more location sensors (such as GPS, for example). Some or all of these sensors are preferably, but not necessarily, integrated into vehicle 100.

[0065] Alternatively or additionally, sensor 30 may include an information system for determining the state of the actuator system. An example of such an information system is a weather information system that determines the current or future state of weather in environment 20.

[0066] For example, using input signal s, policy 60 can, for instance, control a robot that is at least partially autonomous to achieve a predefined target state. Output signal y controls the robot that is at least partially autonomous.

[0067] Preferably, the actuator 10 integrated in the vehicle 100 can be provided by the vehicle 100's brakes, propulsion system, engine, transmission system, or steering mechanism. Preferably, an actuator control command A can be determined to control the actuator (or multiple actuators) 10 so that the vehicle 100 avoids collisions with objects in the environment of at least a partially autonomous robot.

[0068] Preferably, the at least partially autonomous robot is an autonomous vehicle. Possible descriptions of the vehicle's state may include its position, speed, relative distance to other traffic participants, and the coefficient of friction of the road surface (which can vary depending on different environments such as rain, snow, dryness, etc.). Sensors that can measure this state include gyroscopes, angle encoders at the wheels, cameras / lidar / radar, etc. The reward signal from this type of learning will characterize how well the vehicle follows a pre-calculated trajectory (also called a reference trajectory). The reference trajectory can be determined by an optimal planner. The actions of the system can be steering angle, braking, and / or throttle. Preferably, braking pressure or steering angle is output by a strategy, which in particular aims to achieve a minimum braking distance or perform evasive maneuvers, as the (sub)optimal planner would do.

[0069] Note that, in this embodiment, a strategy can be learned to control the dynamics and / or stability of a robot that is at least partially autonomous. For example, if the robot is in a safety emergency, the strategy can, for instance, control the robot to maneuver it out of the emergency by implementing emergency braking. The strategy can then output a value characterizing negative acceleration, which, depending on the value, controls the actor, for example, to interrupt the action with a force associated with the negative acceleration.

[0070] In another embodiment, the at least partially autonomous robot may be provided by another mobile robot (not shown), which may move, for example, by flying, swimming, diving, or walking. The mobile robot may, in particular, be a lawnmower that is at least partially autonomous, or a cleaning robot that is at least partially autonomous.

[0071] In another embodiment, at least partially autonomous robotic operation may be provided by a gardening robot (not shown) that uses sensor 30, preferably an optical sensor, to determine the state of the plants in environment 20. Actuator 10 may be a nozzle for spraying chemicals. Actuator control command A may be determined to cause actuator 10 to spray an appropriate amount of appropriate chemical onto the plants.

[0072] In even further embodiments, at least partially autonomous robots may be provided by household appliances (not shown), such as washing machines, stoves, ovens, microwave ovens, or dishwashers. Sensor 30 (e.g., an optical sensor) can detect the state of the object to be processed by the household appliance. For example, in the case of a washing machine, sensor 30 can detect the state of the clothes inside the washing machine. The actuator control signal A can then be determined based on the detected clothing material.

[0073] Figure 4 The diagram illustrates one embodiment in which a control system 40 controls a manufacturing machine 11 (e.g., a stamping cutter, a cutting machine, or a gun drill) of a manufacturing system 200, which is part of a production line. The control system 40 controls an actuator 10, which in turn controls the manufacturing machine 11.

[0074] Sensor 30 may be provided by an optical sensor that captures, for example, the properties of the manufactured product 12. Strategy 60 may determine the actions to manipulate product 12 based on the state of the manufactured product 12. Actuator 10 controlling manufacturing machine 11 may then be controlled based on the determined state of the manufactured product 12 for subsequent manufacturing steps of product 12. Alternatively, it is conceivable that actuator 10 may be controlled during subsequent manufacturing of product 12, depending on the determined state of the manufactured product 12.

[0075] A preferred embodiment for manufacturing involves the autonomous (disassembly) assembly of certain objects by a robot. The state can be determined by sensors. Preferably, for assembling objects, the state characterizes both the robot manipulator itself and the object to be manipulated. For the robot manipulator, the state can consist of its joint angles and angular velocities, as well as the position and orientation of its end effector. This information can be measured by angle encoders in the joints and gyroscopes that measure the robot joint angular rates. The end effector position and orientation can possibly be derived from kinematic equations. Alternatively, the relative position and orientation with respect to the robot manipulator can be inferred using camera images or lidar scans. The feedback signals for the robot task can, for example, be broken down into different stages of the assembly process. For example, when a pin is inserted into a hole during assembly, the appropriate feedback signal will encode the pin's position and orientation relative to the hole. Typically, the robot system is actuated via a motor at each joint. Therefore, depending on the implementation, the action of the learning algorithm can be the desired torque or a voltage / current directly applied to the motor.

[0076] Figure 5 The illustration shows an embodiment in which the control system 40 controls the automated personal assistant 250. The sensor 30 may be, for example, an optical sensor for receiving video images of gestures from the user 249. Alternatively, the sensor 30 may also be, for example, an audio sensor for receiving voice commands from the user 249.

[0077] The control system 40 then determines an actuator control command A for controlling the automated personal assistant 250. The actuator control command A is determined based on a sensor signal S from sensor 30. The sensor signal S is transmitted to the control system 40. For example, strategy 60 can be configured to determine the action, for instance, based on the state characterizing gesture recognition, which can be determined by an algorithm to identify the gesture made by user 249. The control system 40 can then determine the actuator control command A for transmission to the automated personal assistant 250. It then transmits the actuator control command A to the automated personal assistant 250.

[0078] For example, actuator control command A can be determined based on the identified user gesture recognized by classifier 60. It can then include information that causes the automated personal assistant 250 to retrieve information from a database and output the retrieved information in a format suitable for reception by user 249.

[0079] In another embodiment, it is conceivable that, instead of the automated personal assistant 250, the control system 40 controls a household appliance (not shown) controlled according to an identified user gesture. The household appliance could be a washing machine, stove, oven, microwave oven, or dishwasher.

[0080] Figure 6The diagram illustrates one embodiment in which the control system controls the access control system 300. The access control system can be designed to physically control access. For example, it can include a door 401. Sensor 30 is configured to detect scenarios relevant to determining whether access is permitted. For example, it can be an optical sensor for providing image or video data or for detecting faces.

[0081] Figure 7 The diagram illustrates one embodiment in which the control system 40 controls the monitoring system 400. This embodiment is largely consistent with... Figure 5 The embodiments shown are equivalent. Therefore, only the differences will be described in detail. Sensor 30 is configured to detect the monitored scene. The control system does not necessarily control actuator 10, but rather controls display 10a. For example, machine learning system 60 can determine the classification of the scene, such as whether the scene detected by optical sensor 30 is suspicious. Actuator control signal A transmitted to display 10a can then be configured, for example, to cause display 10a to adjust the displayed content based on the determined classification, such as highlighting objects deemed suspicious by machine learning system 60.

[0082] Figure 8 The diagram illustrates an embodiment of a control system 40 for controlling an imaging system 500, such as an MRI apparatus, an X-ray imaging apparatus, or an ultrasound imaging apparatus. Sensor 30 may be, for example, an imaging sensor. Strategy 60 can then determine actions characterizing the trajectory based on its input states to acquire a record of the imaging system 500.

[0083] The term "computer" encompasses any device used to process predefined computational instructions. These computational instructions can be in the form of software, hardware, or a combination of both.

[0084] To further understand, processes can be implemented not only entirely in software as described, but also in hardware or a hybrid of software and hardware.

Claims

1. A computer-implemented method for learning a policy for an agent, wherein the agent is at least partially autonomous, a robot (100), a manufacturing machine, or a vehicle, wherein the policy receives sensor values ​​characterizing the state of the agent and / or its environment as input and outputs values ​​characterizing control values, wherein the sensor is one of the following: a gyroscope, an accelerometer, a force sensor, a camera, radar, lidar, an angle encoder, or a position sensor, the method comprising the steps of: - Initialization (S1) strategy ( π θ ) and transformation dynamics model ( The transformation dynamics model ( Predict the next state of the environment and / or agent. s t+1 ); - Repeat the following steps until the termination condition is met: - At least record (S2) agent and follow policy ( π θ The interactive scenarios of the environment are added (S3) to the training data. D env A set of ) - Based on training data ( D env ) Optimization (S4) Transformation Dynamics Model ( ), making the transformation dynamics model ( ) depends on the data contained in the training data ( D env The state and actions in the environment are used to predict the next state of the environment; - By following the strategy ( π θ ) Optimize the reward on at least one episode, based on training data ( D env ) and transformation dynamics model ( Optimization (S5) strategy ( π θ )parameter( θ ); Its features Transformation dynamics model ( ) includes the first model representing the learning model of the environment ( g ) and correction model ( d t The correction model ( d t ) is configured to correct the first model ( g The error is ).

2. The method according to claim 1, wherein, In order to optimize the transformation dynamics model ( ), in training data ( D env Optimize the first model on () g After that, select a calibration model that is appropriate for the policy derived from the training data. π θ The selected action, the first model ( g The error is minimized.

3. The method according to claim 1 or 2, wherein, Correction model ( d t ) is optimized so that if the training data is recorded ( D env If the action in the plot is selected as completion, then the transition dynamics model ( The predicted state sequence will be equal to the recorded states of the training data.

4. The method according to claim 3, wherein, By minimizing the correction model ( d t The difference between the output of ) and the following differences is used to select the correction model ( d t ): Recording status of training data ( ) and the first model ( g The difference between the predicted states.

5. The method according to claim 1 or 2, wherein the correction model ( d t ) and the first model ( g Joint optimization.

6. The method according to claim 1 or 2, wherein the correction model ( d t () is state- or time-related, where time represents the time span elapsed since the start of the corresponding plot.

7. The method of claim 6, wherein the environment is deterministic and the correction model is time-dependent.

8. The method of claim 1 or 2, wherein the correction model is a probability function, wherein the probability function is optimized by approximate inference.

9. A method for using a learning strategy that depends on any of the preceding claims ( π θ A computer-implemented method for operating an agent, wherein sensors sense the agent's environment, and wherein the current state is determined based on the sensed environment. s t ), and the learning strategy described therein ( π θ The action to be taken for the agent depends on the current state. a t ), whereby the agent performs the actions described.

10. A machine-readable storage medium (46, 146) having a computer program stored thereon, the computer program being configured to cause a computer to perform the method and all steps of any one of claims 1 to 9 if the computer program is executed by a processor (45, 145).

11. A control system (40) for operating an actuator (10), the control system (40) comprising a strategy (60) trained using the method according to any one of claims 1 to 8, and configured to operate the actuator (10) according to the output of the strategy (60).

Images