Systems and methods for polytopic policy optimization of robust feedback control during learning

Abstract

Learning the dynamics of the system provides a controller for generating policies to control the system. The controller is configured to perform steps including: acquiring measurement data from sensors disposed on the system; providing a nonlinear system model in a memory, the nonlinear system model being represented by known portions of the system's dynamics and unknown portions of the system's dynamics; collecting states by measuring the system's dynamics using the system's sensors based on a nominal policy and a noise term for the states of the system; estimating a sequence of a set of states of the system and a set of control inputs by collecting data for the system, the data including a set of system states, control inputs to be applied, and changes in the system states, each of the control inputs being calculated by the nominal policy and an additional noise term; and learning a polytopic system using the collected data of the system to approximate the unknown portions of the system's dynamics using a linear stochastic regression model; estimating an attractor basin by sampling states in the vicinity of a terminal state and estimating the attractor basin of a terminal controller by supervised learning; and generating a polytopic policy using the estimated polytopic system to advance the system from an initial state to the attractor basin of the terminal controller.

Description

[Technical Field]

[0001] The present invention relates generally to system control, and more particularly to a method and apparatus for robust optimization and feedback control of systems with partially known dynamics during training. [Background technology]

[0002] Although the fields of machine learning and artificial intelligence have made groundbreaking advances over the past decade, the application of these techniques to physical or robotic systems remains limited. For example, most robotic systems still use classical model-based approaches to control complex processes. This can be attributed to several undesirable features that characterize most existing learning approaches, such as a lack of robustness or poor sample efficiency. Since learning models that accurately represent physical mechanics are rare, ensuring robustness during learning for the control of physical systems is crucial.

[0003] Reinforcement learning (RC) is a learning framework for sequential decision-making problems in which an "agent" or decision maker learns a policy that optimizes a long-term reward by interacting with a typically unknown environment. At each iteration or time step, the RL agent obtains evaluation feedback (called reward or cost) on the long-term performance of its actions so that it can improve the performance of subsequent actions.

[0004] Model-free reinforcement learning (RL) methods have gained popularity due to their ability to construct control policies without requiring complex dynamical models. However, the majority of model-free RL algorithms fail to incorporate important system properties, such as robustness to disturbances or even asymptotic stability. Conversely, model-based RL (MBRL) can incorporate such desired properties, but it relies heavily on the underlying dynamical model, and large model mismatches can lead to poor performance, sometimes even catastrophic failure. This problem is highly relevant in many different scenarios where knowing the true dynamics is extremely difficult. This is true for many robotic systems, where robots are expected to interact with novel objects during runtime and therefore cannot have a priori models of the interactions.

[0005] In the policy optimization stage of MBRL techniques, optimization methods must be able to incorporate uncertainty into the learned model and ensure the robustness of the resulting controller to control a partially known system. This is generally difficult to achieve; it is a problem that requires knowledge and representation of the uncertainty present in the system dynamics and the utilization of that uncertainty in the policy optimization step. However, this is necessary for many different applications of learning controllers in areas such as robotics and autonomous driving. Summary of the Invention [Problem to be solved by the invention]

[0006] Most techniques for learning-based controller design suffer from poor performance when implemented in systems because they cannot, in principle, guarantee that the system will converge to the desired system state during training. Previous research in the design of stabilizing controllers for nonlinear systems has used the concept of a controller's basin (or region) of attraction to ensure convergence and stability of the nonlinear system. See, for example, Tedrake, Russ, et al., "LQR-trees: Feedback motion planning via sums-of-squares verification." The International Journal of Robotics Research 29.8 (2010): 1038-1052. Computing the region of attraction of a nonlinear controller may be infeasible during the training phase due to the uncertainty of the learned model. However, it may not be necessary to design a region of attraction that covers the entire state space of the robot. It may be sufficient to design a terminal controller with a known region of attraction. In that case, the control problem can be decomposed into the task of robustly moving the system to the attractor basin of the terminal controller, which then regulates the system's state to the desired terminal state.

[0007] With this understanding, it is important to be able to incorporate the uncertainty in the system dynamics during controller design and design a controller that can reliably move the system from its initial state to the attractor basin of the terminal controller. However, to incorporate the uncertainty during controller design, the uncertainty must be estimated and represented in a way that is suitable for the controller calculation.

[0008] To ensure that a dynamical system always converges to a desired end state, it is necessary to design a terminal controller that ensures the system adjusts to the desired end state once the system reaches the system's terminal basin. In general, estimating the attractor basin for a controller of a nonlinear system is a computationally and algorithmically challenging problem. However, efficient estimation of the attractor basin of a controller can stabilize and quickly learn the desired behavior.

[0009] Therefore, there is a need for robust policy optimization and controller design techniques that utilize an appropriate representation of uncertainty during model training and are capable of designing controllers that robustly control the system to obtain the desired performance during training. [Means for solving the problem]

[0010] An object of some embodiments is to provide systems and methods for data-driven robust optimization of control policies for systems with partially known dynamics during training. Additionally or alternatively, an object of some embodiments is to provide systems and methods capable of learning a polytopic representation of uncertainty for a partially known system. Additionally or alternatively, an object of some embodiments is to provide systems and methods capable of performing robust optimization using a polytopic uncertainty representation with guaranteed local stability to compute a robust polytopic controller. Additionally or alternatively, an object of some embodiments is to provide systems and methods capable of learning a region of attraction for a terminal controller of an underlying system. Additionally or alternatively, an object of some embodiments is to provide systems and methods for controlling an underlying dynamical system with a terminal controller, such as in a region of attraction between the terminal controller and a polytopic controller.

[0011] Some embodiments of the present disclosure provide a novel robust policy optimization mechanism that can address model mismatch by learning the polytope that the residual (unmodeled component of the true dynamics) system is likely to reside in. This polytopic over-approximation of the residual dynamics based on Bayesian regression enables the construction of robust policies by solving tractable semidefinite programs (SDPs).

[0012] According to some embodiments of the present disclosure, a novel computer-implemented method for learning robust policies for model-based reinforcement learning is provided by representing model uncertainty using a Bayesian polytope learned directly from data. The algorithm satisfies input and state constraints. Furthermore, the novel method reduces to a polytopic controller with guaranteed local stability under the assumption of the learned Bayesian polytope. As an example of a robust polytopic controller design, this disclosure demonstrates that estimating a catch basin for the terminal controller leads to better data efficiency and guaranteed stability.

[0013] Such a throw policy is learned from the system's output data, and its role is to propagate the system dynamics toward a specified goal state. At the goal state, a catch policy is employed, leveraging supervised learning to estimate the corresponding capture basin that guarantees (with high probability) that any state can be stabilized to the goal state. Thus, if the throw policy guides the state within this capture basin, the catch policy propels the system toward the goal state. A major advantage of our approach is that it can also incorporate state and input constraints. We demonstrate the proposed algorithm on an inverted pendulum and an underactuated acrobot system.

[0014] Some embodiments are based on the recognition that inaccuracies in a dynamic model of a system lead to compound errors in the predictions of the dynamic system that are used to design a controller for the dynamic system. To achieve a desired behavior of a dynamic system with an inaccurate dynamic model, it is generally necessary to quantify the inaccuracies and then use estimates of the inaccuracies in the controller design so that these inaccuracies can be compensated for during operation.

[0015] Some embodiments of the present disclosure are based on the recognition that for most physical models, predictive models are available using expert knowledge or can be created using a physics engine. These models are generally inaccurate and do not accurately match physical observations from real systems. However, these models can be used to calculate an initial controller for controlling the real system. Some embodiments are based on the recognition that such controllers perform poorly in real systems, but can be used to collect data from the real system. This data can be used to improve the predictive model for the real system.

[0016] Some embodiments of the present disclosure are based on the recognition that data collected from a real system using a controller can be used to calculate the difference in observations between the real system and a model of the system. The difference between the real system and a known model of the system can be learned from the collected data using a machine learning model. Some embodiments of the present disclosure are based on the recognition that the uncertainty of a machine learning model can be quantified in different ways using different expressions for the same thing.

[0017] Some embodiments of the present disclosure are based on the recognition that tools from semi-definite programming (SDP) can be used to design robust feedback controllers for dynamic models with known uncertainties by using linear or polytopic representations of the uncertainties. Such robust controllers can be designed while ensuring local robustness to model uncertainties. Some embodiments of the present disclosure are based on the recognition that a polytopic representation of the uncertainties in a trained model can be obtained using linear Bayesian optimization and predetermined confidence intervals for predictions made by Bayesian regression.

[0018] Some embodiments of the present disclosure are based on the recognition that it may not be possible to modify the dynamic model of a system in parts of the state space that the system cannot reach. When a calculated control trajectory is implemented, the error between the observed state and the calculated nominal trajectory increases with the trajectory horizon. As a result, the system never reaches near the desired end state. Some embodiments are based on this recognition and therefore design a terminal controller that adjusts the system state toward the desired system state and toward the terminal state once the system enters the attractor basin of the terminal controller.

[0019] Some embodiments of the present disclosure are based on the recognition that most dynamical systems require a stable terminal controller to ensure that the dynamical system is tunable and remains at the desired end state of the system. However, apart from the terminal controller, it is necessary to estimate the attractor basin of the terminal controller, where the terminal controller is guaranteed to be stable. Some embodiments of the present disclosure are based on the recognition that terminal controllers for dynamical systems can be designed using model-free or model-based methods, depending on whether a model of the system's end state is known. If a model of the end state is not known, the terminal controller can also be calculated using a model-free approach, such as a proportional, integral, and derivative (PID) controller.

[0020] Some embodiments of the present disclosure are based on the recognition that the attractor basin of a terminal controller can be estimated by sampling states from a neighborhood of a terminal state and training a classifier to predict whether the sampled states can be adjusted to a desired terminal state. Such a learning process can be designed by first sampling states in the neighborhood of the terminal state and then observing the steady-state convergence behavior of the system to the desired terminal state. After collecting data and convergence labels from initial states, a classifier can be trained to predict convergence from any initial state, providing an estimate of the attractor basin of the terminal controller.

[0021] Some embodiments of the present disclosure are based on the recognition that the final controller design involves using a robust SDP-based controller to advance the system states into the controller attractor basin, followed by the terminal controller predicted by the learned classifier.

[0022] According to some embodiments of the present invention, there is provided a controller for generating policies for controlling a system by learning the dynamics of the system. The controller may comprise an interface controller that acquires measurement data from sensors disposed on the system, at least one processor, and a memory that stores computer-implemented instructions, including a model learning module and a policy learning module that, when executed by the at least one processor, cause the controller to perform steps. The steps include providing a nonlinear system model in a memory, the nonlinear system model being represented by known portions of the system's dynamics and unknown portions of the system's dynamics; collecting states by measuring the system's dynamics using sensors of the system based on a nominal policy and a noise term for the system's states; estimating a sequence of a set of system states and a set of control inputs by collecting data for the system, the data including a set of system states, applied control inputs, and changes in the system states, each of the control inputs being calculated by the nominal policy and an additional noise term; and learning a polytopic system using the collected data of the system to approximate the unknown portions of the system's dynamics using a linear stochastic regression model; estimating an attractor basin by sampling initial states near a terminal state and estimating the attractor basin of a terminal controller through supervised learning; and generating a polytopic policy using the estimated polytopic system to advance the system from the initial state to the attractor basin of the terminal controller.

[0023] Additionally, according to some embodiments, a computer-implemented method is provided for generating policies for controlling a system by learning the dynamics of the system. The steps performed according to the computer-implemented method include obtaining measurement data from sensors disposed on the system; providing a nonlinear system model in a memory, the nonlinear system model being represented by known portions of the system's dynamics and unknown portions of the system's dynamics; collecting states by measuring the system's dynamics using the system's sensors based on a nominal policy and a noise term for the system's states; estimating a sequence of a set of system states and a set of control inputs by collecting data for the system, the data including a set of system states, applied control inputs, and changes in the system states, each of the control inputs being calculated by the nominal policy and an additional noise term; and the method further includes learning a polytopic system using the collected data of the system to approximate the unknown portions of the system's dynamics using a linear stochastic regression model; estimating an attractor basin by sampling initial states near a terminal state and estimating the attractor basin of a terminal controller through supervised learning; and generating a polytopic policy using the estimated polytopic system to advance the system from the initial state to the attractor basin of the terminal controller.

[0024] Embodiments of the present disclosure are further described with reference to the accompanying drawings. The illustrated drawings are not necessarily to scale, emphasis generally being placed upon illustrating the principles of embodiments of the present disclosure. [Brief explanation of the drawings]

[0025] [Figure 1A] FIG. 1 illustrates an example of a proposed policy optimization method, according to some embodiments of the present disclosure. [Figure 1B]FIG. 1 illustrates known and unknown portions of system dynamics in the state space of a system and a data collection process according to some embodiments of the present disclosure. [Figure 2] FIG. 1 illustrates a sequence of steps involved in calculating a slow-catch controller using a control method, according to some embodiments of the present disclosure. [Figure 3] FIG. 1 illustrates a system trajectory of an actual system and a known system, according to some embodiments of the present disclosure. [Figure 4] FIG. 1 illustrates the difference between the designed and observed trajectories of a system due to unknowns in the system dynamics. [Figure 5] FIG. 1 illustrates a Bayesian system trained from data collected using real and simulated systems, according to some embodiments of the present disclosure. [Figure 6] FIG. 1 illustrates a sequence of steps involved in learning the region of attraction of a system for a given controller by sampling system states under a controlled system and fitting a classifier model to the sampled data, according to some embodiments of the present disclosure. [Figure 7] FIG. 10 illustrates a suction region of a termination controller of a system according to some embodiments of the present disclosure. [Figure 8] FIG. 1 illustrates the operation of the system during calculation of a Throw-Catch controller, according to some embodiments of the present disclosure. [Figure 9] FIG. 1 illustrates an implementation of a computed robust control on a system where a terminal controller is implemented once the system enters the attractor basin of the terminal controller, according to some embodiments of the present disclosure. [Figure 10A] FIG. 1 is a schematic diagram illustrating a feedback learning system showing different modules of data collection and learning a controller with more data, according to some embodiments of the present disclosure. [Figure 10B]FIG. 1 is a block diagram illustrating a controller including different modules for data collection and learning the controller with more data, according to some embodiments of the present disclosure. [Figure 11] FIG. 1 illustrates an example robotic system showing an example manipulator tracking a desired trajectory from an initial state, according to some embodiments of the present disclosure. [Figure 12A] FIG. 1 illustrates a manipulator of a robotic system manipulating a workpiece, according to some embodiments of the present disclosure. [Figure 12B] 1A-1C illustrate manipulators of a robotic system for manipulating different workpieces, according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] Various embodiments of the present invention are described below with reference to the figures. It should be noted that the figures are not drawn to scale, and that elements having similar structure or function are represented by like reference numerals throughout the figures. It should also be noted that the figures are intended only to facilitate the description of particular embodiments of the present invention. They are not intended to be an exhaustive description of the present invention, nor are they intended to limit the scope of the present invention. Additionally, aspects described in conjunction with a particular embodiment of the present invention are not necessarily limited to that embodiment, but may be practiced in any other embodiment of the present invention.

[0027] According to some embodiments of the present disclosure, robust optimization of machine learning models with uncertainty is important for learning and designing robust controllers for physical systems. Despite significant advances in machine learning and artificial intelligence over the past decade, progress in learning-based control has largely been limited to robotics and classical control systems. Reinforcement learning provides a powerful paradigm for learning high-performance controllers for physical systems that are difficult to model from physics. Such tasks are common in robotics, where a robot must engage in frictional interactions to manipulate its environment or where a robot must react to changes in the environment based on sensory observations. Modeling such tasks is challenging because mathematical models are unavailable or the parameters of known models are uncertain.

[0028] According to some embodiments of the present disclosure, most model-free reinforcement learning algorithms lack the desirable theoretical guarantees of stability and asymptotic convergence that classical model-based control approaches can provide. Meanwhile, the performance of model-based reinforcement learning approaches is determined by the accuracy of the model available for control. While learning highly accurate predictive models can be challenging, recent machine learning models are capable of learning the uncertainties of predictive models. These uncertainties capture the noise observed in the evolution of the system, resulting from operational noise, observation errors, or errors in the known physics of the underlying system. These uncertainties can be used by a properly designed optimizer to design a robust controller that adjusts the system state to approach a desired trajectory and ultimately converge to a desired end state. However, to utilize the uncertainty associated with the learned model, the uncertainty must be properly represented and made available to the optimizer. Thus, designing a robust controller for a partially known system involves three key steps: uncertainty estimation, uncertainty representation, and satisfaction of uncertainty constraints during optimization.

[0029] In some embodiments of the present disclosure, the controller learning problem for an unknown system is divided into two parts. FIG. 1A illustrates an example of a proposed policy optimization method 100 according to some embodiments of the present disclosure. In this case, the dynamics of the real system are described by known dynamics and a Bayes polytope learned using data collected from the real system. The system operates with the terminal controller when it reaches the attracting region of the system's terminal controller and uses the slow controller when it is outside of that region. The first part of the controller advances the system state to the known terminal region of the terminal controller. This part is known as the slow controller 130. Note that the system always starts from a fixed initial state and must be adjusted to the desired terminal state (or goal state). The second part of the controller 140 is the terminal controller that is activated when the system enters the attractor basin of the terminal controller. The complete learning process is denoted as Throw-Catch Polytopic Policy Optimization (TCPPO). An example trajectory 114 of the system of 110 is a sequence of states 112 that the system passes through to reach a terminal region 113, starting from an initial state 111. Some embodiments of the present disclosure learn a polytopic representation 122 of the system dynamics that can be used to design a robust controller. Some embodiments of the present disclosure learn a robust polytopic controller 130 to propel the system into the terminal region 113. Some embodiments of the present disclosure learn a terminal controller 140 to ensure convergence of the system state to a desired terminal state.

[0030] Some exemplary embodiments disclosed herein relate to the problem of learning unknown dynamics of a dynamical system. FIG. 1B illustrates a diagram illustrating known and unknown portions of system dynamics 180 in the system's state space and a data collection process 161, according to some embodiments of the present disclosure. In this case, the system evolution 180 is composed of known portion 160 and unknown portion 170. The unknown portion of dynamics 170 may result from unmodeled phenomena in complex systems, changes in some submodules of the system, or uncertainties in the parameters of the dynamical model. Learning the residual between the true dynamics and the known dynamics can be achieved using a machine learning module that predicts the residual given the current state and control inputs to the system.

[0031] Some embodiments of the present disclosure implement a method for learning a controller for a system with partially known dynamics. Figure 2 illustrates a series of steps involved in computing a slow-catch controller using a control method 200, which may be a computer-implemented method (or program module) 200 for collecting data consisting of system states, target states, and a known system model 160. The computer-implemented method 200 uses a known system to accept an initial search policy 212 to collect data 213 from the real system. This collected data is used to learn 214 a Bayesian model of the unknown portions of the system dynamics. This learned model is used to compute 215 a robust policy for navigating the system to a terminal controller terminal region using semidefinite programming. In certain embodiments of the present disclosure, a terminal controller is designed 216 and its attractor basin is estimated 217 using sampling and machine learning methods.

[0032]

number

[0033]

number

[0034]

number

[0035] Some embodiments disclosed in the present invention relate to the problem of estimating the uncertainty in the prediction of learned residual system dynamics. By estimating the uncertainty in the learned residual dynamics, a robust controller can be designed (generated) by considering the worst-case residual dynamics between the true system dynamics and the known system dynamics. The uncertainty associated with the learned residual dynamics can be estimated by modeling the residuals as a stochastic function of the input variables. A probabilistic machine learning model can then be used to represent the uncertainty in the prediction of the residual dynamics.

[0036]

number

[0037]

number

[0038]

number

[0039]

number

[0040]

number

[0041] FIG. 4 illustrates the difference between the designed and observed trajectories of a system 400 due to unknowns in the system dynamics, and FIG. 5 illustrates a Bayesian system 500 trained from data collected using real and simulated systems, according to some embodiments of the present disclosure.

[0042]

number

[0043] The system 400 uses this learned polytopic representation of the system dynamics to generate / compute a robust polytopic policy for guiding the system to a terminal set 413. Given the unknowns in the dynamics and the imperfections in the estimates obtained from the machine learning model, a non-robust controller will be unable to steer the system 400 from the initial state 410 to the desired terminal set 413 of terminal states 412. A controller that does not account for the uncertainties will deviate, resulting in an unreachable trajectory 414 in the terminal region.

[0044]

number

[0045]

number

[0046]

number

[0047]

number

[0048] To tune the state of a dynamical system to a desired terminal state, some embodiments of the present disclosure design a terminal controller that can guarantee convergence of the system state to the terminal state. Such a portion of the controller is called a catch controller. However, any tuning controller can only guarantee convergence within its region of attraction. Estimating the region of attraction for a general nonlinear system is numerically and algorithmically challenging. To implement the robust controller described in this disclosure, it is necessary to estimate the region of attraction of the terminal controller. Based on this recognition, some embodiments of the present disclosure estimate the region of attraction 700 of the terminal controller. The region of attraction (or attractor basin) 700 of the controller is the set of states of the dynamical system that can be advanced to the terminal state 710 by applying the controller via any trajectory 720. When the controller acts on any state in this set, the system state in this set always converges to the state x T Converge to 710.

[0049]

number

[0050] 6 is a diagram illustrating a sequence of steps (computer-implemented method) 600 involved in learning the region of attraction of a system for a given controller by sampling system states under a controlled system and fitting a classifier model to the sampled data, according to some embodiments of the present disclosure. Additionally, FIG. 7 is a diagram illustrating the region of attraction of a terminal controller 700 of the system, according to some embodiments of the present disclosure.

[0051]

number

[0052]

number

[0053] FIG. 8 is a diagram illustrating the operation of a policy 800 used by the system in calculating a throw-catch controller according to some embodiments of the present disclosure.

[0054]

number

[0055] Some embodiments of the present disclosure provide a computer-implemented method for implementing a slow-catch policy for a system with model mismatch. This embodiment accepts a system state during operation (810). A trained classifier model can determine whether the current state is within the region of attraction of a terminal (catch) controller (820). If the state is within the terminal region (840), the terminal controller is implemented in the system. Otherwise, a robust polytopic controller (or slow controller) is implemented (830). Note that, as Equation (10) suggests, the final control policy for the system is a hybrid control policy that uses different controllers in different portions of the state space.

[0056] FIG. 9 is a diagram illustrating the implementation of a computed robust control 900 on a system where a terminal controller is implemented once the system enters the attractor basin of the terminal controller, according to some embodiments of the present disclosure.

[0057] The system behavior can be seen in the computed robust control 900, where the designed nominal trajectory is 911 from the initial state 910. Due to unknowns in the system dynamics, the observed system trajectory 912 deviates from the designed trajectory. However, the robust polytopic controller drives the system into the attractor basin of the terminal controller 930. The system is driven to the terminal state 920 using the terminal controller via trajectory 913.

[0058] FIG. 10A is a schematic diagram of a feedback learning system according to some embodiments of the present disclosure, illustrating different modules of data collection and learning the controller with more data.

[0059] In this case, a feedback learning system is used to design and control a motor controller 1000. The system accepts commands (desired goals) 1010, which are provided to a controller 1020. The controller uses a nominal policy to collect survey data 1031. This data is used by a TCP / IP learning method 1040 to calculate a new controller, which is then sent back to the controller 1020, which updates the control laws for controlling the machine. This process is repeated until convergence.

[0060] FIG. 10B is a block diagram illustrating a controller (robot control system) 50 including different modules for data collection and controller training using more data, according to some embodiments of the present disclosure. The robot control system 50 is configured to control the robot's actuator system 1203. In some cases, the robot controller 100B included in the controller 50 may be hardware configured to connect to an actuator controller 150B for controlling the robot's motion. The robot controller 100B, including the model learning module 1300B and the policy learning module 1400B, is connected to the actuator controller 150B and the robot, and to sensors 1201 that measure the robot's motion and are disposed on the robot to detect the states of the robot's individual actuators. Furthermore, the robot controller 100B generates policy parameters based on the measured data of the robot's motion, and the robot controller 100B provides the policy parameters to the robot's actuator controller 150B to update the actuator controller's policy unit 151B.

[0061] The robot controller 100B, which may be a hardware circuit, may include an interface controller 110B, a processor 120, and a memory unit 130B. The processor 120 may be one or more processor units, and the memory unit 130B may be a memory device, a data storage device, or the like. The interface controller 110B may be an interface circuit and may include an analog / digital (A / D) converter and a digital / analog (D / A) converter for signal / data communication with the robot's sensors 1201 and the motion controller 150B. Furthermore, the interface controller 110B may include memory for storing data used by the A / D converter or D / A converter. The sensors 1201 are located on the joints of the robot (robot arm(s)) or on the picking mechanism (e.g., fingers) to measure the robot's statistics. The robot includes an actuator controller (device / circuit) 150B, which includes a policy unit 151B that generates operation parameters for controlling the robot system 1203, which controls the robot arm, handling mechanism, or combination of arm and handling mechanism 1203-1, 1203-2, 1203-3, 1203-#n depending on the number of joints or handling fingers. For example, the sensors 1201 may include acceleration sensors, positioning sensors, torque sensors, angle sensors, global positioning system (GPS) devices, cameras, depth cameras, distance cameras, range scanners, etc. for measuring the motion state of the robot.

[0062] The interface controller 110B is also connected to sensors 1201 mounted on the robot that measure / acquire the state of the robot's motion. In some cases, if the actuators are electric motors, the actuator controller 150B may control individual electric motors that drive the angle of the robot arm or the handling of an object by the handling mechanism. In some cases, the actuator controller 150B may control the rotation of individual motors located on the arm in response to policy parameters generated from the policy learning module 1400B to smoothly accelerate or safely decelerate the robot's motion. Furthermore, depending on the design of the object handling mechanism, the actuator controller 150B may control the length of the actuator in response to policy parameters generated from the policy learning module 1400B.

[0063] The memory unit 130B can store computer-executable program modules including a model learning module 1300B and a policy learning module 1400B. The processor 120 is configured to execute steps of the program modules 1300B and 1400B. In this case, the steps may include offline modeling using the model learning module 1300B to generate offline learning states based on an operational state (motion state) of the robot and measurement states from the sensor 1201. The steps further include providing the offline states to the policy learning module 1400B to generate policy parameters, and updating the policy 151B of the robot's motion controller 150B to operate the actuator system 1203 based on the policy parameters.

[0064] For example, the measurement data can consist of the angular positions of the joints of a manipulator arm, the pose of a robot's end effector, or the pose of an object manipulated by the manipulator arm. The dynamics of the system can be defined by the relationship between the change in angular position of the joints resulting from the application of torques to individual joints of the robot. It can also be defined by the relationship between the change in the pose of the object due to the application of forces to the object by the manipulator. The state of the system can be represented by the joint angular positions and velocities to which the object being manipulated by the robot can be subjected. The state of the system can be measured using encoders or a vision system (camera, depth camera, range scanner, etc.) that can be designed to track objects in the robot's environment. The set of control inputs are the torques applied by motors at individual joints to move the joint or end effector position. The attractor basin of a terminal controller can be defined by the set of system states near the terminal state that can be adjusted by the terminal controller. For a robotic manipulator manipulating an object, the terminal state is the desired pose of the object. The terminal controller can be calculated using classical control methods that ensure the object's state converges to the desired state. The attractor basin of a terminal controller is the set of states near the terminal state for which the terminal controller can guarantee convergence.

[0065] FIG. 11 illustrates an example of a robotic system 1100 showing a manipulator tracking a desired trajectory from an initial state, according to some embodiments of the present disclosure. FIGS. 12A and 12B illustrate a manipulator of the robotic system 1100 manipulating one workpiece 1210 and a different workpiece 1220, according to some embodiments of the present disclosure. The proposed controller 50 is configured to control the movement of the manipulator arm of the robotic system 1100 with an additional unknown load 1101. The manipulator controller 50 sends control commands based on sensor feedback from sensors to control the load to follow the desired trajectory 1120. However, due to the unknown load, there is a slight error in the observed trajectory 1130 when compared to the designed trajectory 1120. This problem is solved by using a TCPPO controller in the manipulator controller 50, allowing the robot to accurately move the different unknown objects 1210 and 1220.

Claims

1. 1. A controller for generating policies for controlling a system by learning the dynamics of the system, the controller comprising: an interface controller that acquires system status from sensors located on the system; at least one processor; and a memory storing computer-implemented instructions including a model learning module and a policy learning module that, when executed by the at least one processor, cause the at least one processor to perform steps of: providing a nonlinear system model in the memory, the nonlinear system model being represented by known portions of the dynamics of the system and unknown portions of the dynamics of the system; calculating the system state at a next time using the current system state based on a nominal policy that represents a known portion of the dynamics of the system and a noise term added to the nominal policy; and estimating a sequence of the set of system states and a set of control inputs by collecting data of the system, the data including the set of system states, applied control inputs, and changes in the system states, each of the applied control inputs being calculated by the nominal policy and the noise term, the step further comprising: using the collected data of the system to learn a polytopic representation for rewriting the nonlinear system model into a polytopic system to generate a polytopic policy for progressing the system from an initial state to an attractor basin of a terminal controller; sampling the system states in the vicinity of a terminal state and estimating the attractor basin of the terminal controller of the system by supervised learning; generating the polytopic policy to advance the system from the initial state to the attractor basin of the terminal controller.

2. 2. The controller of claim 1, further comprising implementing a hybrid control policy comprising a robust polytopic policy that steers the system into the attractor basin of the terminal controller and uses the terminal controller to steer the system to a desired terminal state.

3. The controller of claim 1 , wherein the controller further learns residual dynamics between the system and the known system model of the dynamics of the system.

4. The controller of claim 1 , wherein the controller further collects data on residual dynamics as a difference in state trajectories for a given control trajectory between the system and the known system model.

5. The controller of claim 1 , wherein the controller further uses a probabilistic machine learning model of residual dynamics using Bayesian optimization.

6. The controller of claim 1 , wherein the controller further uses the polytopic system and semidefinite programming to construct a robust policy.

7. The controller of claim 1 , wherein the termination controller at the termination state of the system is generated to advance the system to a desired termination state.

8. The controller of claim 1 , wherein the controller further uses machine learning based classification techniques to estimate the attractor basin of the terminal controller.

9. 1. A computer-implemented method for generating policies for controlling a system by learning the dynamics of the system, comprising: acquiring system status from sensors located on the system; providing a nonlinear system model in a memory, the nonlinear system model being represented by known portions of the dynamics of the system and unknown portions of the dynamics of the system; calculating the system state at a next time using the current system state based on a nominal policy that represents a known portion of the dynamics of the system and a noise term added to the nominal policy; and estimating a sequence of the set of system states and a set of control inputs by collecting data for the system, the data including the set of system states, control inputs to be applied, and changes in the system states, each of the control inputs being calculated by the nominal policy and the noise term, the method further comprising: using the collected data of the system to learn a polytopic representation for rewriting the nonlinear system model into a polytopic system to generate a polytopic policy for progressing the system from an initial state to an attractor basin of a terminal controller; sampling the system states in the vicinity of a terminal state and estimating the attractor basin of the terminal controller of the system by supervised learning; generating the polytopic policy to advance the system from the initial state to the attractor basin of the terminal controller.

10. 10. The computer-implemented method of claim 9, further comprising implementing a hybrid control policy comprising a robust polytopic policy that steers the system into the attractor basin of the terminal controller and uses the terminal controller to steer the system to a desired terminal state.

Images