Systems and methods for polytopic policy optimization of robust feedback control during learning

The system addresses the challenge of uncertain system dynamics by using polytopic representations and Bayesian regression to design controllers that robustly guide systems to a desired state, enhancing stability and efficiency in robotic and physical systems.

JP2024541746AActive Publication Date: 2024-11-11MITSUBISHI ELECTRIC CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024550931
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-10
Filing Date
2022-09-09
Publication Date
2024-11-11
Estimated Expiration
2042-09-09

AI Technical Summary

Technical Problem

Existing learning-based controller designs for robotic systems and physical systems lack robustness and stability due to uncertainties in system dynamics, leading to poor performance and potential catastrophic failures, especially when interacting with unknown environments.

Method used

A system and method for robust policy optimization using polytopic representations of uncertainty, incorporating Bayesian regression and semidefinite programming to design controllers that ensure convergence to a desired end state by learning the attractor basin of the end controller.

Benefits of technology

Ensures stable and efficient control of systems with partially known dynamics by accounting for uncertainties, improving data efficiency and guaranteeing local stability through polytopic controllers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541746000001_ABST
    Figure 2024541746000001_ABST
Patent Text Reader

Abstract

Learning the dynamics of the system provides a controller for generating policies to control the system. The controller is configured to perform steps including: acquiring measurement data from sensors disposed on the system; providing a nonlinear system model in a memory, the nonlinear system model being represented by known portions of the system's dynamics and unknown portions of the system's dynamics; collecting states by measuring the system's dynamics using the system's sensors based on a nominal policy and a noise term for the states of the system; estimating a sequence of a set of states of the system and a set of control inputs by collecting data for the system, the data including a set of system states, control inputs to be applied, and changes in the system states, each of the control inputs being calculated by the nominal policy and an additional noise term; and learning a polytopic system using the collected data of the system to approximate the unknown portions of the system's dynamics using a linear stochastic regression model; estimating an attractor basin by sampling states in the vicinity of a terminal state and estimating the attractor basin of a terminal controller by supervised learning; and generating a polytopic policy using the estimated polytopic system to advance the system from an initial state to the attractor basin of the terminal controller.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates generally to system control, and more particularly to a method and apparatus for robust optimization and feedback control of systems with partially known dynamics during training. [Background technology]

[0002] Although the fields of machine learning and artificial intelligence have made breakthrough improvements over the past decade, the application of these techniques is limited when it comes to physical or robotic systems. For example, most robotic systems still use classical model-based approaches to control complex processes. This can be attributed to several undesirable features that characterize most existing learning approaches, such as lack of robustness or poor sample efficiency. Since (learning) models that accurately represent physical mechanics are rare, it is important to ensure robustness during learning for the control of physical systems.

[0003] Reinforcement learning (RC) is a learning framework for sequential decision problems in which an "agent" or decision maker learns a policy that optimizes long-term rewards by interacting with a typically unknown environment. At each iteration or time step, the RL agent obtains evaluation feedback (called reward or cost) on the long-term performance of its actions so that it can improve the performance of subsequent actions.

[0004] Model-free reinforcement learning (RL) methods have become widespread due to their ability to construct control policies without the need for complex dynamical models. However, the majority of model-free RL algorithms fail to incorporate important system properties, such as robustness to disturbances or even asymptotic stability. Conversely, model-based RL (MBRL) can incorporate such desired properties, but it relies heavily on the underlying dynamical model, and large model mismatches can lead to poor performance, sometimes even catastrophic failure. This problem is highly relevant in many different scenarios where the true dynamics are very difficult to know. This is true for many robotic systems, where the robot is expected to interact with new objects during execution and therefore cannot have a priori models of the interactions.

[0005] In the policy optimization stage of MBRL techniques, the optimization method must allow for the incorporation of uncertainty in the learned model and ensure the robustness of the resulting controller to control a partially known system. This is generally difficult to achieve; it is a problem that requires knowledge and representation of the uncertainty present in the system dynamics and the exploitation of the uncertainty in the policy optimization step. However, this is necessary for many different applications to learning controllers in areas such as robotics, autonomous driving, etc. Summary of the Invention [Problem to be solved by the invention]

[0006] Most techniques for learning-based controller design suffer from poor performance when implemented in systems because they cannot in principle guarantee that the system will converge to the desired system state during learning. Prior work in designing stabilizing controllers for nonlinear systems has used the concept of a controller's basin (or region) of attraction to guarantee convergence and stability of the nonlinear system. See, for example, Tedrake, Russ, et al. "LQR-trees: Feedback motion planning via sums-of-squares verification." The International Journal of Robotics Research 29.8 (2010): 1038-1052. Computing the region of attraction of a nonlinear controller may be infeasible during the learning phase due to uncertainties in the learned model. However, it may not be necessary to design a region of attraction that covers the entire state space of the robot. It may be sufficient to design a terminal controller with a known region of attraction. The control problem can then be decomposed into the task of robustly moving the system to the attractor basin of the terminal controller, which can then regulate the state of the system to the desired terminal state.

[0007] With this understanding, it is important to be able to incorporate the uncertainties in the system dynamics during controller design and design a controller that can reliably move the system from its initial state to the attractor basin of the terminal controller. However, to incorporate the uncertainties during controller design, the uncertainties must be estimated and represented in a way that is suitable for the controller calculations.

[0008] To ensure that a dynamical system always converges to a desired end state, it is necessary to design a terminal controller that ensures the adjustment of the system to the desired end state once the system reaches the system's terminal basin. In general, estimating the attractor basin for a controller of a nonlinear system is a very challenging problem, both computationally and algorithmically. However, an efficient estimation of the attractor basin of the controller can stabilize and quickly learn the desired behavior.

[0009] Therefore, there is a need for robust policy optimization and controller design techniques that utilize an appropriate representation of the uncertainty during model training and that can design a controller that robustly controls the system to obtain the desired performance during training. [Means for solving the problem]

[0010] It is an object of some embodiments to provide a system and method for data-driven robust optimization of a control policy of a system with partially known dynamics during learning. Additionally or alternatively, it is an object of some embodiments to provide a system and method capable of learning a polytopic representation of uncertainty for a partially known system. Additionally or alternatively, it is an object of some embodiments to provide a system and method capable of performing robust optimization using a polytopic uncertainty representation with guaranteed local stability to compute a robust polytopic controller. Additionally or alternatively, it is an object of some embodiments to provide a system and method capable of learning a region of attraction of a terminal controller of an underlying system. Additionally or alternatively, it is an object of some embodiments to provide a system and method for controlling an underlying dynamical system with a terminal controller, such as in a region of attraction between the terminal controller and a polytopic controller.

[0011] Some embodiments of the present disclosure provide a novel robust policy optimization mechanism that can address model mismatch by learning the polytope that the residual (unmodeled component of the true dynamics) system is likely to reside in. This polytopic over-approximation of the residual dynamics based on Bayesian regression allows the construction of robust policies by solving tractable semidefinite programs (SDPs).

[0012] According to some embodiments of the present disclosure, a novel computer-implemented method for learning robust policies for model-based reinforcement learning is provided by representing model uncertainty with a Bayesian polytope learned directly from data. The algorithm satisfies input and state constraints. Furthermore, the novel method reduces to a polytopic controller with guaranteed local stability under the assumption of the learned Bayesian polytope. As an example of designing a robust polytopic controller, the present disclosure describes how estimating a catch basin for the terminal controller leads to better data efficiency and guaranteed stability.

[0013] Such a throw policy is learned from the system's output data and its role is to propagate the system dynamics towards a specified goal state. At the goal state, we employ a catch policy and leverage supervised learning to estimate the corresponding capture basin, where any state is guaranteed (with high probability) to be stabilizable to the goal state. Thus, if the throw policy leads the state within this capture basin, the catch policy propels the system towards the goal state. A major advantage of our approach is that it can also incorporate state and input constraints. We demonstrate the proposed algorithm on an inverted pendulum and an underactuated acrobot system.

[0014] Some embodiments are based on the recognition that inaccuracies in a dynamic model of a system lead to compound errors in the predictions of the dynamic system that are used to design a controller for the dynamic system. To achieve a desired behavior of a dynamic system with an inaccurate dynamic model, it is generally necessary to quantify the inaccuracies and then use estimates of the inaccuracies in the controller design in order to be able to compensate for these inaccuracies during operation.

[0015] Some embodiments of the present disclosure are based on the recognition that for most physical models, predictive models are available using expert knowledge or can be created using a physics engine. These models are generally inaccurate and do not exactly match physical observations from real systems. However, these models can be used to calculate initial controllers for controlling the real systems. Some embodiments are based on the recognition that such controllers perform poorly in real systems, but can be used to collect data from the real systems. This data can be used to improve the predictive models of the real systems.

[0016] Some embodiments of the present disclosure are based on the recognition that data collected from a real system using a controller can be used to calculate the difference in observations between the real system and a model of the system. The difference between the real system and a known model of the system can be learned from the collected data using a machine learning model. Some embodiments of the present disclosure are based on the recognition that the uncertainty of a machine learning model can be quantified in different ways using different expressions for the same thing.

[0017] Some embodiments of the present disclosure are based on the realization that tools from Semi-Definite Programming (SDP) can be used to design robust feedback controllers for dynamic models with known uncertainties by using linear or polytopic representations of the uncertainties. Such robust controllers can be designed while ensuring local robustness to model uncertainties. Some embodiments of the present disclosure are based on the realization that a polytopic representation of the uncertainties in a trained model can be obtained using linear Bayesian optimization and a given confidence interval for the predictions by Bayesian regression.

[0018] Some embodiments of the present disclosure are based on the recognition that it may not be possible to modify the dynamic model of the system in parts of the state space that the system cannot reach. When implementing a calculated control trajectory, the error between the observed state and the calculated nominal trajectory grows with the trajectory horizon. As a result, the system never reaches near the desired end state. Some embodiments are based on this recognition and therefore design a terminal controller that adjusts the system state to the desired system state and to the terminal state once the system enters the attractor basin of the terminal controller.

[0019] Some embodiments of the present disclosure are based on the recognition that most dynamical systems require a stable terminal controller to ensure that the dynamical system is tunable and remains at the desired end state of the system. However, apart from the terminal controller, it is necessary to estimate the attractor basin of the terminal controller where the terminal controller is guaranteed to be stable. Some embodiments of the present disclosure are based on the recognition that the terminal controller of a dynamical system can be designed using model-free or model-based methods, depending on whether the model of the terminal state of the system is known. If the model of the terminal state is not known, the terminal controller can also be calculated using a model-free approach such as a proportional, integral and derivative (PID) controller.

[0020] Some embodiments of the present disclosure are based on the recognition that the attractor basin of a terminal controller can be estimated by sampling states from the neighborhood of a terminal state and training a classifier that predicts whether the sampled states can be adjusted to a desired terminal state. Such a learning process can be designed by first sampling states in the neighborhood of a terminal state and then observing the steady-state convergence behavior of the system to the desired terminal state. After collecting data and convergence labels from the initial states, a classifier can be trained to predict the convergence from any initial state to provide an estimate of the attractor basin of the terminal controller.

[0021] Some embodiments of the present disclosure are based on the recognition that the final controller design uses a robust SDP-based controller to advance the system states into the controller attractor basin, and then uses the terminal controller predicted by the learned classifier.

[0022] According to some embodiments of the present invention, there is provided a controller for generating policies to control a system by learning the dynamics of the system. The controller may comprise an interface controller that acquires measurement data from sensors disposed on the system, at least one processor, and a memory storing computer-implemented instructions including a model learning module and a policy learning module that, when executed by the at least one processor, cause the controller to perform steps. The steps include providing in a memory a nonlinear system model represented by known parts of the system's dynamics and unknown parts of the system's dynamics; collecting states by measuring the system's dynamics using sensors of the system based on a nominal policy and a noise term for the states of the system; estimating a sequence of a set of states of the system and a set of control inputs by collecting data for the system, the data including a set of system states, applied control inputs, and changes in the system states, each of the control inputs being calculated by the nominal policy and an additional noise term; and the steps further include learning a polytopic system using the collected data of the system to approximate the unknown parts of the system's dynamics using a linear stochastic regression model; estimating an attractor basin by sampling initial states in a neighborhood of a terminal state and estimating the attractor basin of a terminal controller by supervised learning; and generating a polytopic policy using the estimated polytopic system to advance the system from the initial state to the attractor basin of the terminal controller.

[0023] Further, according to some embodiments, a computer-implemented method is provided for generating policies for controlling a system by learning the dynamics of the system. The steps performed according to the computer-implemented method include obtaining measurement data from sensors disposed on the system; providing a nonlinear system model in a memory, the nonlinear system model being represented by known parts of the system's dynamics and unknown parts of the system's dynamics; collecting states by measuring the system's dynamics using the system's sensors based on a nominal policy and a noise term for the states of the system; estimating a sequence of a set of states of the system and a set of control inputs by collecting data for the system, the data including a set of system states, control inputs to be applied, and changes in the system states, each of the control inputs being calculated by the nominal policy and an additional noise term; and the method further includes learning a polytopic system using the collected data of the system to approximate the unknown parts of the system's dynamics using a linear stochastic regression model; estimating an attractor basin by sampling initial states in a neighborhood of a terminal state and estimating the attractor basin of a terminal controller by supervised learning; and generating a polytopic policy using the estimated polytopic system to advance the system from the initial state to the attractor basin of the terminal controller.

[0024] Embodiments of the present disclosure are further described with reference to the accompanying drawings. The illustrated drawings are not necessarily to scale, with emphasis generally being placed upon illustrating the principles of embodiments of the present disclosure. [Brief description of the drawings]

[0025] [Figure 1A] FIG. 2 illustrates an example of a proposed policy optimization method according to some embodiments of the present disclosure. [Figure 1B]FIG. 2 illustrates a diagram showing known and unknown parts of the system dynamics in the state space of the system and the data collection process according to some embodiments of the present disclosure. [Diagram 2] FIG. 2 illustrates a sequence of steps involved in the calculation of a slow-catch controller using a control method, according to some embodiments of the present disclosure. [Diagram 3] FIG. 2 illustrates a system trajectory of an actual system and a known system, according to some embodiments of the present disclosure. [Figure 4] FIG. 1 illustrates the difference between the designed and observed trajectories of a system due to unknowns in the system dynamics. [Diagram 5] FIG. 1 illustrates a Bayesian system trained from data collected using real and simulated systems, according to some embodiments of the present disclosure. [Figure 6] FIG. 1 illustrates a sequence of steps involved in learning the region of attraction of a system for a given controller by sampling system states under a controlled system and fitting a classifier model to the sampled data, in accordance with some embodiments of the present disclosure. [Figure 7] FIG. 13 illustrates a suction region of a termination controller of a system according to some embodiments of the present disclosure. [Figure 8] FIG. 2 illustrates the operation of the system during calculation of a Throw-Catch controller, according to some embodiments of the present disclosure. [Figure 9] FIG. 1 illustrates an implementation of a computed robust control on a system where a terminal controller is implemented once the system enters the attractor basin of the terminal controller, according to some embodiments of the present disclosure. [Figure 10A] FIG. 1 is a schematic diagram illustrating a feedback learning system showing different modules of data collection and learning a controller with more data according to some embodiments of the present disclosure. [Figure 10B]FIG. 2 is a block diagram illustrating a controller including different modules for data collection and learning the controller with more data, according to some embodiments of the present disclosure. [Figure 11] FIG. 1 illustrates an example robotic system showing an example manipulator tracking a desired trajectory from an initial state, according to some embodiments of the present disclosure. [Figure 12A] FIG. 1 illustrates a manipulator of a robotic system manipulating a workpiece according to some embodiments of the present disclosure. [Figure 12B] 1A-1D illustrate manipulators of a robotic system for manipulating different workpieces according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Various embodiments of the present invention are described below with reference to the figures. It should be noted that the figures are not drawn to scale, and elements having similar structures or functions are represented by similar reference numerals throughout the figures. It should also be noted that the figures are intended only to facilitate the description of specific embodiments of the present invention. They are not intended to be an exhaustive description of the present invention, nor are they intended to limit the scope of the present invention. In addition, aspects described in conjunction with a specific embodiment of the present invention are not necessarily limited to that embodiment, but may be practiced in any other embodiment of the present invention.

[0027] According to some embodiments of the present disclosure, robust optimization of machine learning models with uncertainty is important for learning and designing robust controllers in physical systems. Despite the tremendous progress in machine learning and artificial intelligence over the past decade, progress in learning-based control has barely extended to robotics and classical control systems. Reinforcement learning provides a powerful paradigm for learning high-performance controllers for physical systems that are difficult to model from physics. Such tasks are common in robotics, where a robot needs to engage in frictional interactions to manipulate its environment, or where a robot needs to react to changes in the environment based on sensory observations. Modeling such tasks is difficult because there is no mathematical model or the parameters of the known model are uncertain.

[0028] According to some embodiments of the present disclosure, most model-free reinforcement learning algorithms lack the desirable theoretical guarantees of stability and asymptotic convergence that classical model-based control approaches can provide. On the other hand, the performance of model-based reinforcement learning approaches depends on the accuracy of the models available for control. Although it can be difficult to learn highly accurate predictive models, recent machine learning models allow learning of the uncertainties of predictive models. These uncertainties capture the noise observed in the evolution of the system, due to actuation noise, observation errors, or errors in the known physics of the underlying system. These uncertainties can be used by a properly designed optimizer to design a robust controller to adjust the system state to approach the desired trajectory and ultimately converge to the desired end state. However, in order to use the uncertainties associated with the learned model, the uncertainties need to be properly represented and made available to the optimizer. Thus, there are three key steps in the design of a robust controller in a partially known system: uncertainty estimation, uncertainty representation, and satisfaction of uncertainty constraints during optimization.

[0029] In some embodiments of the present disclosure, the controller learning problem for an unknown system is split into two parts. FIG. 1A illustrates an example of a proposed policy optimization method 100 according to some embodiments of the present disclosure. In this case, the dynamics of the real system are described by the known dynamics and a Bayes polytope learned using data collected from the real system. The system operates with the terminal controller when it reaches the attractor region of the terminal controller of the system, and uses the slow controller when it is outside of it. The first part of the controller drives the system state to the known terminal region of the terminal controller. This part is known as the slow controller 130. Note that the system always starts from a fixed initial state and must be adjusted to the desired terminal state (or goal state). The second part of the controller 140 is the terminal controller that is activated when the system enters the attractor basin of the terminal controller. The complete learning process is denoted as Throw-Catch Polytopic Policy Optimization (TCPPO). An example trajectory 114 of the system of 110 is a sequence of states 112 that the system goes through to reach a terminal region 113, starting from an initial state 111. Some embodiments of the present disclosure learn a polytopic representation 122 of the system dynamics that can be used to design a robust controller. Some embodiments of the present disclosure learn a robust polytopic controller 130 to propel the system into the terminal region 113. Some embodiments of the present disclosure learn a terminal controller 140 to ensure convergence of the system states to a desired terminal state.

[0030] Some exemplary embodiments disclosed herein relate to the problem of learning unknown dynamics of a dynamical system. FIG. 1B is a diagram illustrating known and unknown parts of system dynamics 180 in the state space of the system and a data collection process 161 according to some embodiments of the present disclosure. In this case, the evolution 180 of the system is composed of known parts 160 and unknown parts 170. The unknown parts of the dynamics 170 may be due to unmodeled phenomena in complex systems, changes in some submodules of the system, or uncertainties in the parameters of the dynamical model. Learning the residual between the true dynamics and the known dynamics can be achieved using a machine learning module that predicts the residual given the current state and the control inputs to the system.

[0031] Some embodiments of the present disclosure implement a method for learning a controller for a system with partially known dynamics. FIG. 2 illustrates a series of steps involved in computing a slow-catch controller using a control method 200, which may be a computer-implemented method (or program module) 200 for collecting data consisting of system states, target states, and a known system model 160. In the computer-implemented method 200, a known system is used to accept an initial search policy (212) to collect data (213) from the real system. This collected data is used to learn (214) a Bayesian model of the unknown parts of the system dynamics. This learned model is used to compute (215) a robust policy for navigating the system into a terminal region of the terminal controller using semidefinite programming. In certain embodiments of the present disclosure, a terminal controller is designed (216) and its attractor basin is estimated (217) using sampling and machine learning techniques.

[0032]

number

[0033]

number

[0034]

number

[0035] Some embodiments disclosed in the present invention relate to the problem of estimating the uncertainty in the prediction of the learned residual system dynamics. By estimating the uncertainty in the learned residual dynamics, a robust controller can be designed (generated) by considering the worst-case residual dynamics between the true system dynamics and the known system dynamics. The uncertainty associated with the learned residual dynamics can be estimated by modeling the residuals as a probabilistic function of the input variables. Then, a probabilistic machine learning model can be used to represent the uncertainty in the prediction of the residual dynamics.

[0036]

number

[0037]

number

[0038]

number

[0039]

number

[0040]

number

[0041] FIG. 4 illustrates the difference between the designed and observed trajectories of a system 400 due to unknowns in the system dynamics, and FIG. 5 illustrates a Bayesian system 500 trained from data collected using real and simulated systems, according to some embodiments of the present disclosure.

[0042]

number

[0043] The system 400 uses this learned polytopic representation of the system dynamics to generate / compute a robust polytopic policy for guiding the system to a terminal set 413. Given the unknowns in the dynamics and the imperfections in the estimates obtained from the machine learning model, a non-robust controller will not be able to steer the system 400 from the initial state 410 to the desired terminal set 413 of terminal states 412. A controller that does not account for the uncertainties will deviate, resulting in an unreachable trajectory 414 in the terminal region.

[0044]

number

[0045]

number

[0046]

number

[0047]

number

[0048] To regulate the state of the dynamical system to a desired terminal state, some embodiments of the present disclosure design a terminal controller that can guarantee the convergence of the system state to the terminal state. Such a portion of the controller is called a catch controller. However, any regulating controller can only guarantee convergence within its region of attraction. Estimating the region of attraction for general nonlinear systems is difficult both numerically and algorithmically. To implement the robust controllers described in this disclosure, it is necessary to estimate the region of attraction of the terminal controller. Based on this recognition, some embodiments of the present disclosure estimate the region of attraction 700 of the terminal controller. The region of attraction (or attractor basin) 700 of the controller is the set of states of the dynamical system that can be advanced to a terminal state 710 by applying the controller via any trajectory 720. When the controller acts on any state in this set, the system state in this set will always be in the state x T Converges to 710.

[0049]

number

[0050] Figure 6 is a diagram illustrating a sequence of steps (computer-implemented method) 600 involved in learning the region of attraction of a system for a given controller by sampling system states under a controlled system and fitting a classifier model to the sampled data, according to some embodiments of the present disclosure. Additionally, Figure 7 is a diagram illustrating the region of attraction of a terminal controller 700 of the system, according to some embodiments of the present disclosure.

[0051]

number

[0052]

number

[0053] FIG. 8 is a diagram illustrating the operation of a policy 800 used by the system in calculating a throw-catch controller in accordance with some embodiments of the present disclosure.

[0054]

number

[0055] Some embodiments of the present disclosure provide a computer-implemented method for implementing a slow catch policy for a system with model mismatch. The embodiment accepts a system state during operation (810). A trained classifier model can determine whether the current state is within the region of attraction of a terminal (catch) controller (820). If the state is within the terminal region (840), the terminal controller is implemented in the system. Otherwise, a robust polytopic controller (or slow controller) is implemented (830). Note that, as equation (10) suggests, the final control policy for the system is a hybrid control policy that uses different controllers in different parts of the state space.

[0056] FIG. 9 is a diagram illustrating an implementation of a computed robust control 900 on a system where a terminal controller is implemented once the system enters the attractor basin of the terminal controller, according to some embodiments of the present disclosure.

[0057] The system behavior can be seen in the computed robust control 900, where the designed nominal trajectory is 911 from the initial state 910. Due to unknowns in the system dynamics, the observed system trajectory 912 deviates from the designed trajectory. However, with the robust polytopic controller, the system is driven into the attractor basin of the terminal controller 930. The system is driven to the terminal state 920 with the terminal controller via trajectory 913.

[0058] FIG. 10A is a schematic diagram of a feedback learning system according to some embodiments of the present disclosure, illustrating different modules of data collection and learning the controller with more data.

[0059] In this case, a feedback learning system is used to design and control a motor controller 1000. The system accepts commands (which are the desired goals) 1010 that are provided to a controller 1020. The controller uses a nominal policy to collect survey data 1031. This data is used by the TCPPO learning method 1040 to compute a new controller. This is then sent back to the controller 1020 which updates the control laws for controlling the machine. This process is repeated until convergence.

[0060] FIG. 10B is a block diagram illustrating a controller (robot control system) 50 including different modules of data collection and learning of the controller with more data according to some embodiments of the present disclosure. The robot control system 50 is configured to control the actuator system 1203 of the robot. In some cases, the robot controller 100B included in the controller 50 can be a hardware configured to connect to an actuator controller 150B for controlling the robot's movement. The robot controller 100B including the model learning module 1300B and the policy learning module 1400B is connected to the actuator controller 150B and the robot and to a sensor 1201 that measures the robot's movement, the sensor 1201 is disposed on the robot to detect the state of the individual actuators of the robot. Furthermore, the robot controller 100B generates policy parameters based on the measured data of the robot's movement, and the robot controller 100B provides the policy parameters to the actuator controller 150B of the robot to update the policy unit 151B of the actuator controller.

[0061] The robot controller 100B, which may be a hardware circuit, may include an interface controller 110B, a processor 120, and a memory unit 130B. The processor 120 may be one or more processor units, and the memory unit 130B may be a memory device, a data storage device, etc. The interface controller 110B may be an interface circuit and may include an analog / digital (A / D) converter and a digital / analog (D / A) converter for signal / data communication with the robot's sensor 1201 and the motion controller 150B. Furthermore, the interface controller 110B may include a memory for storing data used by the A / D converter or the D / A converter. The sensors 1201 are located on the joints of the robot (robot arm(s)) or on the picking object mechanism (such as fingers) to measure the statistics of the robot. The robot includes an actuator controller (device / circuit) 150B, which includes a policy unit 151B for generating operation parameters for controlling the robot system 1203, which controls the robot arms, handling mechanisms, or combinations of arms and handling mechanisms 1203-1, 1203-2, 1203-3, 1203-#n depending on the number of joints or handling fingers. For example, the sensors 1201 may include acceleration sensors, positioning sensors, torque sensors, angle sensors, global positioning system (GPS) devices, cameras, depth cameras, distance cameras, range scanners, etc. for measuring the motion state of the robot.

[0062] The interface controller 110B is also connected to sensors 1201 mounted on the robot that measure / acquire the state of the robot's motion. In some cases, if the actuators are electric motors, the actuator controller 150B may control individual electric motors that drive the angle of the robot arm or the handling of the object by the handling mechanism. In some cases, the actuator controller 150B may control the rotation of individual motors arranged on the arm in order to smoothly accelerate or safely decelerate the robot's motion in response to the policy parameters generated from the policy learning module 1400B. Furthermore, depending on the design of the object handling mechanism, the actuator controller 150B may control the length of the actuator in response to the policy parameters generated from the policy learning module 1400B.

[0063] The memory unit 130B can store computer executable program modules including a model learning module 1300B and a policy learning module 1400B. The processor 120 is configured to execute steps of the program modules 1300B and 1400B. In this case, the steps may include offline modeling using the model learning module 1300B to generate offline learning states based on the operation state (motion state) of the robot and the measurement state from the sensor 1201. The steps further include providing the offline states to the policy learning module 1400B to generate policy parameters, and updating the policy 151B of the robot's motion controller 150B to operate the actuator system 1203 based on the policy parameters.

[0064] For example, the measurement data may consist of the angular positions of the joints of a manipulator arm, the pose of the end effector of a robot, or the pose of an object manipulated by a manipulator arm. The dynamics of the system may be defined by the relationship of the change in angular positions of the joints resulting from the application of torques to the individual joints of the robot. It may also be defined by the relationship of the change in the pose of the object due to the application of forces to the object by the manipulator. The state of the system may be represented by the joint angular positions and velocities to which the object being manipulated by the robot may be subjected. The state of the system may be measured using encoders or using a vision system (camera, depth camera, range scanner, etc.) that may be designed to track objects in the robot's environment. The set of control inputs are the torques applied by the motors at the individual joints to move the joints or the position of the end effector. The attractor basin of the terminal controller may be defined by the set of system states near the terminal state that can be adjusted by the terminal controller. For a robot manipulator manipulating an object, the terminal state is the desired pose of the object. The terminal controller may be calculated using classical control methods that ensure that the state of the object converges to the desired state. The attractor basin of a terminal controller is the set of states near the terminal state for which the terminal controller can guarantee convergence.

[0065] FIG. 11 is a diagram illustrating an example of a robotic system 1100 showing a manipulator tracking a desired trajectory from an initial state according to some embodiments of the present disclosure. FIG. 12A and FIG. 12B are diagrams illustrating a manipulator of a robotic system 1100 manipulating one workpiece 1210 and a different workpiece 1220 according to some embodiments of the present invention. The proposed controller 50 is configured to control the movement of the manipulator arm of the robotic system 1100 with an additional unknown load 1101. The manipulator controller 50 sends control commands based on the sensor feedback of the sensors to control the load to follow the desired trajectory 1120. However, there is a small error in the observed trajectory 1130 due to the unknown load when compared to the designed trajectory 1120. This problem is solved by using a TCPO controller in the manipulator controller 50 so that the robot can accurately move the different unknown objects 1210 and 1220.

Claims

1. 1. A controller for generating a policy for controlling a system by learning dynamics of the system, comprising: an interface controller for acquiring measurement data from sensors disposed on the system; At least one processor; and a memory storing computer-implemented instructions including a model learning module and a policy learning module that, when executed by the at least one processor, cause the controller to perform steps including: providing a nonlinear system model to the memory, the nonlinear system model being represented by known parts of the dynamics of the system and unknown parts of the dynamics of the system; collecting said state by measuring the dynamics of the system using the sensors of the system based on a nominal policy and a noise term for the state of the system; and estimating a sequence of a set of states and a set of control inputs of the system by collecting data of the system, the data including a set of system states, control inputs to be applied, and changes in system states, each of the control inputs to be applied being calculated by the nominal policy and the additional noise term, the step further comprising: training a polytopic system using the collected data of the system to approximate the unknown portions of the dynamics of the system using a linear stochastic regression model; estimating an attractor basin of a terminal controller by sampling states in a neighborhood of the terminal state and estimating the attractor basin of the terminal controller by supervised learning; and generating a polytopic policy using the estimated polytopic system to advance the system from an initial state to the attractor basin of the terminal controller.

2. 2. The controller of claim 1, further comprising implementing a hybrid control policy comprising a robust polytopic policy that steers the system into an attractor basin of the termination controller and uses the termination controller to steer the system to a desired terminal state.

3. The controller of claim 1 , wherein the controller further generates a search policy using the known system model of the system.

4. The controller of claim 1 , wherein the controller further collects data from the system by implementing a search policy on the system and collecting state trajectories and input trajectories.

5. The controller of claim 1 , wherein the controller further learns residual dynamics between the system and the known system model of the system's dynamics.

6. The controller of claim 1 , wherein the controller further collects data regarding residual dynamics as a difference in state trajectories for a given control trajectory between the system and the known system model.

7. The controller of claim 1 , wherein the controller further employs a probabilistic machine learning model of residual dynamics using Bayesian optimization.

8. The controller of claim 1 , wherein the controller further estimates a polytopic system of residual dynamics using a Bayesian model and a predetermined confidence interval.

9. The controller of claim 1 , further comprising: generating a robust polytopic controller using the polytopic system and semidefinite programming.

10. The controller of claim 1 , wherein the termination controller at the terminal state of the system is generated to advance the system to the desired terminal state.

11. The controller of claim 1 , wherein the controller further uses a machine learning based classification technique to estimate the attractor basin of the terminal controller.

12. 1. A computer-implemented method for generating policies for controlling a system by learning the dynamics of the system, comprising: acquiring measurement data from sensors located on the system; providing a nonlinear system model to the memory, the nonlinear system model being represented by known parts of the dynamics of the system and unknown parts of the dynamics of the system; collecting said state by measuring the dynamics of the system using the sensors of the system based on a nominal policy and a noise term for the state of the system; and estimating a sequence of a set of states and a set of control inputs of the system by collecting data of the system, the data including a set of system states, control inputs to be applied, and changes in system states, each of the control inputs being calculated by the nominal policy and the additional noise term, the method further comprising: training a polytopic system using the collected data of the system to approximate the unknown portions of the dynamics of the system using a linear stochastic regression model; estimating an attractor basin of a terminal controller by sampling initial states in a neighborhood of the terminal state and estimating the attractor basin of the terminal controller by supervised learning; and generating a polytopic policy using the estimated polytopic system to advance the system from an initial state to the attractor basin of the terminal controller.

13. 13. The computer-implemented method of claim 12, further comprising implementing a hybrid control policy comprising a robust polytopic policy that steers the system into an attractor basin of the terminal controller and uses the terminal controller to steer the system to a desired terminal state.

14. The computer-implemented method of claim 12 , wherein the controller further generates a search policy using the known system model of the system.

15. 13. The computer-implemented method of claim 12, wherein the controller further collects data from the system by implementing a search policy on the system and collecting state trajectories and input trajectories.

16. The computer-implemented method of claim 12 , wherein the controller further learns residual dynamics between the system and the known system model of the system's dynamics.

17. 13. The computer-implemented method of claim 12, wherein the controller further collects data regarding residual dynamics as a difference between state trajectories for a given control trajectory between the system and the known system model.

18. 13. The computer-implemented method of claim 12, wherein the controller further uses a probabilistic machine learning model of residual dynamics using Bayesian optimization.

19. 13. The computer-implemented method of claim 12, wherein the controller further estimates a polytopic system of residual dynamics using a Bayesian model and a predetermined confidence interval.

20. 13. The computer-implemented method of claim 12, wherein the controller further uses the polytopic system and semidefinite programming to generate a robust polytopic controller.

21. 13. The computer-implemented method of claim 12, wherein the termination controller at the terminal state of the system is generated to advance the system to the desired terminal state.

22. The computer-implemented method of claim 12 , wherein the controller further estimates the attractor basin of the terminal controller using a machine learning based classification technique.

Citation Information

Patent Citations

  • Graph generation device, method and program

    JP2013222388A

  • Controller having neural network and improved stability

    JP2021152898A

  • Model Predictive Control with Uncertainties

    US20160246277A1

  • System and Method for Robust Optimization for Trajectory-Centric ModelBased Reinforcement Learning

    US20210178600A1