Systems and methods for robust optimization of trajectory-centric model-based reinforcement learning

Abstract

A controller for optimizing a local control policy of a system for trajectory-centric reinforcement learning is provided. The controller includes performing the following steps: learning a stochastic predictive model of the system using a set of data collected during trial-and-error experiments performed using an initial random control policy; estimating associated mean predictions and uncertainty; determining a local set of deviations of the system from a nominal system state using the learned stochastic system model at a current time step using a control input; determining a system state with a worst-case deviation; determining a gradient of a robustness constraint; using a nonlinear programming to provide and solve a robust policy optimization problem to obtain a system trajectory and simultaneously stabilize the local policy; updating control data according to the solved optimization problem; and outputting the updated control data via an interface.

Description

Technical Field

[0001] This invention generally relates to systems and methods for simultaneously optimizing local policies and control trajectories in trajectory-centric reinforcement learning. Background Technology

[0002] Reinforcement learning (RL) is a learning framework for handling sequential decision-making problems, in which an "agent" or decision-maker learns a policy that optimizes long-term rewards by interacting with a (unknown) environment. At each step, the RL agent receives evaluative feedback (called reward or cost) about its performance on its actions, enabling it to improve (maximize or minimize) its performance on subsequent actions.

[0003] Generally, global learning and optimization of arbitrary nonlinear systems can be extremely challenging both computationally and algorithmically. However, many systems perform many tasks that are trajectory-centric, making local learning and optimization highly efficient in terms of data. Trajectory-centric control can be challenging for nonlinear systems due to the time-varying nature of the controller. Deviations from the planned trajectory during operation are common for real-world systems due to model inaccuracies or noise in observations or actuation. Machine learning methods allow learning and then predicting uncertainties in the evolution of the controlled trajectory. From a control perspective, the goal is to design local state-dependent policies that can stabilize the controlled trajectory using the learned uncertainty model. Most techniques cannot stabilize the desired control trajectory using knowledge of uncertainties present in the system model.

[0004] Another desired approach is to design the trajectory and its corresponding stable policy simultaneously. This naturally involves a trade-off between the optimality of the control trajectory and its stability. Intuitively, in this setting, the policy optimization algorithm will avoid regions of the state space that might be more difficult to control, thus allowing the design of a robust, optimal trajectory-centric controller by leveraging uncertainties in the model. Most current techniques perform these two steps (trajectory design and controller synthesis) separately, thus failing to utilize this knowledge of model uncertainties.

[0005] Given the aforementioned facts and challenges, there is a need for better strategy optimization methods that can utilize uncertain statistical models of physical systems and leverage the structure of these models to achieve robust performance of the system across a wide range of tasks. Summary of the Invention

[0006] Recent research has led to significant success for these algorithms in various fields, such as computer games. In trajectory-centric RL, the goal is to optimize a policy that can successfully perform a task starting from the system's initial state and guide the system to the desired final state. Trajectory-centric methods have the advantage of being able to learn faster because they learn local predictive models and use them to optimize the policy in the local neighborhood of the system.

[0007] Reinforcement learning algorithms can be broadly categorized into two types—model-based methods and model-free methods. Model-based reinforcement learning (MBRL) techniques are generally considered data-efficient because they learn a predictive model for the system that is task-independent. The learned model is then used to synthesize the system's policy using stochastic control methods. However, these methods are often difficult to train, resulting in poorly performing policies. There are several reasons for this low performance, one of the key challenges being the presence of noise in the data collected during the learning process, meaning that the predicted model of the system estimated during this period is inherently stochastic. As a result, an incorrect model can drive the optimization algorithm into unstable parts of the system's state space, and the learning process may then diverge. Another challenging aspect of MBRL is that the estimated model may have varying degrees of uncertainty in different regions of the state space; therefore, subsequent policy optimization steps should leverage this structure of the learned statistical model to achieve optimal performance. Most policy optimization techniques either ignore or fail to incorporate this information during policy optimization.

[0008] The advantage of MBRL is that the predictive models estimated during learning are task-independent, thus enabling them to be used for multiple tasks and making them more efficient in learning across multiple tasks. Therefore, MBRL allows the reuse of learned models to compute policies for different tasks. As a result, MBRL has the potential to learn effective policies for many physical systems where collecting large amounts of data to optimize policies can be very expensive.

[0009] According to some embodiments of the present invention, uncertain statistical models of physical systems are used to perform policy optimization to leverage the structure of these models to achieve robust performance of these systems across a wide range of tasks. Furthermore, some embodiments provide controllers and methods capable of handling arbitrary nonlinear systems by efficiently executing computational and associated learning algorithms. Since many tasks that many systems need to perform are trajectory-centric, some embodiments of the present invention can provide highly efficient local learning and optimization of data, thereby improving computer performance.

[0010] Some embodiments of this disclosure are based on the understanding that a computer-implemented learning method is provided for optimizing the control strategy of a control system. Examples of such systems may include mechanical systems such as HVAC systems, factory automation systems, robotic systems, and high-performance induction motors. In this case, the method may include: receiving the state of a system operating according to a specific task strategy; initializing the control strategy to an initial strategy for performing the task (e.g., a known partial model of the system and the task); collecting data using the current control strategy, including tuples of the current state, actions, and the next state; estimating a statistical prediction model of the system; estimating the predictions under any state and control pair, and the uncertainties associated with the predictions; estimating the local set around the nominal system state to which the system needs to be stabilized; computing the gradients of robustness constraints on the parameters that can be used to optimize the locally stable strategy; computing the gradients of the system dynamics; and determining a trajectory-centric control strategy by solving a nonlinear optimization procedure using the gradients of the robust constraints to minimize the trajectory cost and the locally stable strategy.

[0011] Some implementations provide methods and systems for performing trajectory-centered tasks for a system in the presence of uncertainty in the statistical model estimated during learning. A controller for optimizing a local control strategy for a system using trajectory-centered reinforcement learning may include: an interface configured to receive data comprising tuples of system states, control data, and state transitions measured by sensors; a memory storing a processor-executable program including a stochastic prediction learning model for generating a nominal state and control trajectory (open-loop trajectory) as a function of time steps over a desired time range in response to system task commands received via the interface; a control strategy including a machine learning method algorithm and an initial stochastic control strategy; and a local strategy for adjusting deviations along the nominal trajectory; at least one processor configured to: learn a stochastic prediction model of the system using a set of data collected during trial-and-error experiments performed using the initial stochastic control strategy; estimate the average prediction and uncertainty associated with the stochastic prediction model; and perform trajectory-centered learning. The trace-centered controller synthesis problem is formulated to simultaneously compute the nominal trajectory along with feedforward control and steady-state time-invariant feedback control. At the current time step, using the control input, a local set of deviations from the nominal system state is determined using a learned stochastic system model. Within this local set of deviations, the system state with the worst-case deviation from the nominal system state is identified. The gradient of the robustness constraints is determined by computing the first derivative of the robustness constraints at the system state with the worst-case deviation. The optimal system state trajectory, feedforward control input, and local time-invariant feedback strategy for adjusting the system state to the nominal trajectory are determined by minimizing the cost of the state control trajectory while satisfying both state and input constraints. A robust strategy optimization problem is provided and solved using nonlinear programming. The control data is updated based on the solved optimization problem, and the updated control data is output via an interface.

[0012] Furthermore, another embodiment of the present invention provides a computer-implemented method (control system) for controlling a system by optimizing a control strategy. The computer-implemented method is provided for optimizing local control strategies for a system using trajectory-centered reinforcement learning. The method may include the following steps: learning a stochastic prediction model of the system using a set of data collected during trial-and-error experiments performed with an initial stochastic control strategy; estimating the average prediction and uncertainty associated with the stochastic prediction model; formulating a trajectory-centered controller synthesis problem to simultaneously compute the nominal trajectory along with feedforward control and steady-state time-invariant feedback control; determining a local set of deviations of the system from the nominal system state using the learned stochastic system model at the current time step when using control inputs; identifying the system state with the worst-case deviation from the nominal system state within the local set of system deviations; determining the gradient of robust constraints by computing the first derivative of the robust constraints in the system state with the worst-case deviation; determining the optimal system state trajectory, feedforward control input, and local time-invariant feedback strategy to adjust the system state to the nominal trajectory by minimizing the cost of the state control trajectory while satisfying the state and input constraints; providing and solving a robust strategy optimization problem using nonlinear programming; updating the control data based on the solved optimization problem; and outputting the updated control data via an interface.

[0013] The system may include: an interface configured to receive system state and task commands via sensors; a memory storing a computer-executable program including an initializer, a policy collector, an estimator, an agent and a policy updater, a program for estimating a stochastic prediction model of the system, a program for computing robust constraints and their gradients, and a program for solving a nonlinear optimization problem to obtain an optimal trajectory and a stable policy; and a processor connected to the memory, configured to initialize a control policy as an initial guess for performing the task (e.g., a known partial model of the system and the task); collect data using the current control policy, including tuples of the current state, action, and next state; estimate a statistical prediction model of the system; estimate predictions under any state and control pair, and the uncertainties associated with the predictions; estimate the local set of nominal system states to which the system needs to be stabilized; compute gradients of robust constraints that can be used to optimize the locally stable policy; compute gradients of the system dynamics; and determine a trajectory-centric control policy by solving a nonlinear optimization program using the gradients of robust constraints to minimize the trajectory cost and the locally stable policy.

[0014] The embodiments disclosed herein will be further explained with reference to the accompanying drawings. The drawings are not necessarily drawn to scale, but rather focus on illustrating the principles of the embodiments disclosed herein. Attached Figure Description

[0015] [ Figure 1A ]

[0016] Figure 1A This is a diagram illustrating a control system (controller) for controlling a mechanical system (device, e.g., robot) based on a quasi-Newtonian trust region strategy optimization according to an embodiment of the present invention;

[0017] [ Figure 1B ]

[0018] Figure 1B This is a diagram illustrating the data processing flow using a controller (control system) according to an embodiment of the present invention;

[0019] [ Figure 2 ]

[0020] Figure 2 This is a diagram illustrating an example of a system based on the proposed robust trajectory-centric controller optimization according to an embodiment of the present invention;

[0021] [ Figure 3 ]

[0022] Figure 3 The figure illustrates the general concept of reinforcement learning according to embodiments of the present disclosure;

[0023] [ Figure 4 ]

[0024] Figure 4 This is a diagram illustrating a concept of model-based reinforcement learning according to an embodiment of the present disclosure;

[0025] [ Figure 5A ]

[0026] Figure 5A This is a graph illustrating the estimation of random input-output relationships using a Gaussian process and the set of uncertainties that can be estimated using confidence intervals according to embodiments of this disclosure;

[0027] [ Figure 5B ]

[0028] Figure 5B This is a graph illustrating the estimation of random input-output relationships using a Gaussian process and the set of uncertainties that can be estimated using confidence intervals according to embodiments of the present disclosure;

[0029] [ Figure 6A ]

[0030] Figure 6A This is a diagram illustrating the concept of the dynamic evolution of a system using a stochastic process according to an embodiment of the present disclosure;

[0031] [ Figure 6B ]

[0032] Figure 6B This is a diagram illustrating the concept of using a stabilization strategy to stabilize a controlled trajectory during the design of a trajectory-centric controller according to an embodiment of the present disclosure;

[0033] [ Figure 7 ]

[0034] Figure 7 This is a flowchart illustrating the different steps involved in solving a nonlinear optimization procedure used to solve a trajectory-centered controller according to an embodiment of the present disclosure;

[0035] [ Figure 8 ]

[0036] Figure 8 This is a flowchart illustrating different steps in estimating the gradients for robust constraints in trajectory-centered controller synthesis according to embodiments of the present disclosure;

[0037] [ Figure 9 ]

[0038] Figure 9 This is a diagram illustrating an example of the stable behavior of a design controller according to an embodiment of the present disclosure;

[0039] [ Figure 10A ]

[0040] Figure 10A This is a diagram illustrating an example of a control trajectory and stabilization strategy obtained during an optimization process according to an embodiment of the present disclosure;

[0041] [ Figure 10B ]

[0042] Figure 10B This is a diagram illustrating an example of a control trajectory and stabilization strategy obtained during an optimization process according to an embodiment of the present disclosure;

[0043] [ Figure 10C ]

[0044] Figure 10C This is a diagram illustrating an example of a control trajectory and stabilization strategy obtained during an optimization process according to an embodiment of the present disclosure;

[0045] [ Figure 11 ]

[0046] Figure 11 This is a diagram illustrating examples of the statistical behavior of a controlled system according to embodiments of the present disclosure under various levels of disturbances in the injected system;

[0047] [ Figure 12 ]

[0048] Figure 12 This is a diagram illustrating an example of a ball-and-beam system;

[0049] [ Figure 13 ]

[0050] Figure 13 This is a figure illustrating a performance comparison example of a ball-beam system using the proposed trajectory-centered controller in open-loop control according to an embodiment of the present disclosure. Detailed Implementation

[0051] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with a sufficient description for implementing one or more exemplary embodiments. Various modifications in the function and arrangement of the elements may be contemplated without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims.

[0052] Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary details. In other instances, known processes, structures, and techniques may be shown without non-essential details to avoid obscuring the embodiments. Furthermore, similar reference numerals and symbols in the various figures refer to similar elements.

[0053] Furthermore, various implementations can be described as processes depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. When the operations of a process are completed, the process can terminate, but may have additional steps not discussed or included in the diagram. Moreover, not all operations in any particular described process can occur in all implementations. A process can correspond to a method, function, process, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the function can correspond to the function returning to the calling function or the main function.

[0054] Furthermore, implementations of the disclosed subject matter can be carried out, at least partially, manually or automatically. They can be performed, or at least assisted by, using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, for manual or automatic implementation. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks can be stored in a machine-readable medium. The processor can then perform the necessary tasks.

[0055] Figure 1A A block diagram of a control system (controller) 100 according to some embodiments of the present invention is shown. System 100 may include a human-machine interface (HMI) 110 with input / output (I / O) interfaces connectable to a keyboard 111 and a pointing device / medium 112, a 3D sensor 116, a processor 120, a storage device 130 storing algorithms to be executed using memory 140 and processor 120, a network interface controller 150 (NIC) connectable to a network 190 including a local area network and an internet network (not shown), a display interface 160 connected to a display device 165, and an imaging interface 170 connectable to an imaging device 175. The NIC may include a transceiver (not shown) to perform wireless communication for receiving commands and status measurement data. The HMI 110 with I / O interfaces may include analog-to-digital and digital-to-analog converters. The HMI 110 with I / O interfaces may also include a wireless communication interface that can communicate with other 3D point cloud display systems or other computers via a wireless internet connection or wireless local area network capable of constructing multiple 3D point clouds. System 100 may include a power supply (not shown). The power supply may be a battery that is rechargeable from an external power source (not shown) via I / O interface 110. Depending on the application, the power supply may be located externally to system 100.

[0056] System 100 can receive electronic text / image documents 195, including voice data, via network 190 connected to NIC 150. Storage device 130 includes algorithm module 131 and robot control algorithm 132. Algorithm module 131 includes a model uncertainty estimator, a robust constraint gradient estimator, and a nonlinear optimization program solver stored as program code data in storage 130. Algorithm 131, including model learning, robust constraint gradient estimator, and nonlinear program optimizer, can be stored on a computer-readable recording medium (not shown), allowing processor 120 to execute the algorithms of models 131-132 and the multi-model encoder-decoder 200 by loading the algorithms from the medium. Furthermore, the fixed-point device / medium 112 may include a module for reading and executing programs stored on the computer-readable recording medium.

[0057] Figure 1BThis is a diagram illustrating a data processing flow using a controller (control system) 100 according to an embodiment of the present invention. Some embodiments of this disclosure are based on the understanding that the controller 100 for machine 103 can be designed to implement a specific command or task 101. The controller 100 is computer software that sends control signals to machine 103, which implements control signals, and the state of the machine is observed by an observer 164. Some embodiments of this disclosure are based on the understanding that a model-based reinforcement learning algorithm 105 can be used to learn the optimal policy of the controller 100 using operational data 104 obtained from the machine. For a specific task or command, the controller is initialized with an initial policy for determining the control signals based on the current state of the machine. The state of the machine can be observed by an observer. The control signals are then implemented on the machine and the state transitions of the machine are observed. This tuple of the current state, control signals, and the next state is the data used by the learning algorithm to calculate a new policy for executing the command or task. This process is then repeated until the policy converges to a stable policy (i.e., a policy that does not change as more data is collected).

[0058] Figure 2 This is a diagram illustrating an example of a robustly optimized control system based on the proposed trajectory-centered RL according to an embodiment of the present invention. In the following description, the control system is depicted as an example applied to robot system 200; however, it should be noted that the control system and computer-implemented method according to the present invention are not limited to robot systems. Some systems or computer-implemented methods according to the present invention can be introduced into systems including mechanical systems (e.g., HVAC systems), factory automation systems, robot systems, and high-performance induction motors, etc.

[0059] Figure 2The robot system 200 may include a robot manipulator equipped with sensors for state measurement and a computer instrumentation system for storing data and controlling the manipulator. The manipulator may include several rigid links 217, 215, 213 and joints 318, 216, 214. The manipulator is controlled using a robot control system that receives commands or tasks provided to the system from an external source. Examples of commands or tasks may be using the manipulator to grasp object 212 or performing an assembly operation using object 211 held by gripper 212. The robot control system sends control signals 270 to the manipulator. Control signals 270 may be torques to be applied to each joint 318, 216, 214 of the manipulator and opening / closing gripper 212. Sensors are used to measure the state of the robot system. These sensors may include encoders at the joints 318, 216, 214, cameras 210 that can observe the robot's environment, and tactile sensors 219 that can be attached to the grippers 212. State measurements from sensors are sent to data input / output unit 220, which stores the data received from the sensors. Some embodiments of this disclosure initialize the robot control system using an initial control strategy for the desired task. A multi-stage control strategy is implemented on the robot system, and state measurement data from sensors is collected in data input / output unit 220. This data is then used by software including computer program 230 to update the current strategy of the robot system. The program for updating the strategy includes a model update system 240, which may be a Gaussian process. The updated model is then transmitted to controller update system 250, which may be a robust trajectory-centric strategy optimizer. Controller update system 250 then transmits the updated strategy to robot control system 260. This process is repeated until the optimal strategy for robot control is reached.

[0060] Figure 3 This diagram illustrates a general concept of reinforcement learning according to embodiments of the present disclosure. Reinforcement learning (RL) is a learning framework for handling sequential decision-making problems, in which an “agent” 330 or decision-maker learns a policy that optimizes long-term rewards (or costs) by interacting with a (unknown) environment 310. At each step, the RL agent receives evaluative feedback (referred to as reward or cost) 350 regarding the performance of its action 340, as well as observations of the environment, enabling it to improve (maximize or minimize) the performance of subsequent actions. The policy of the system is defined as a function that takes the current state of the system as input and then generates an action as output. The optimal policy is the policy that generates the optimal output for each state of the system, where optimality is measured relative to the expected reward (or cost) for a given task.

[0061] Reinforcement learning algorithms are broadly classified into two categories: model-free and model-based. As the name suggests, model-free RL algorithms do not use the system's state transition probabilities to learn the policy. In these types of algorithms, the system's policy is parameterized as a function approximator. The policy parameters are then optimized directly by maximizing the expected reward for performing the desired task or by estimating a task-related value function. These algorithms are typically sample-inefficient, meaning they require large amounts of training data to optimize the system's policy. Furthermore, learning using these systems is task-dependent, as the optimization is performed to maximize the expected reward for a specific task.

[0062] Among different types of RL algorithms, MBRL is considered to be more data-efficient. In model-based RL algorithms, an initial (potentially stochastic) control policy is used to collect state transition data from the system. This state transition data is then used to learn a forward predictive model of the system. It is important to note that the learning of the predictive model is independent of any task and therefore also independent of any reward structure. Once a good estimate of the system's forward dynamics is obtained, the system's dynamic model is used to synthesize a controller capable of performing the desired task. The task can be performed using either a controlled trajectory or a state-dependent complete policy. A controlled trajectory is a set of control inputs implemented on the system that can start from known initial conditions to achieve the desired state for performing the task. Note that such a controller is not state-dependent but time-indexed. A state-dependent complete policy can be independent of the system's initial state and therefore can be used to perform the desired task starting from any initial state of the system.

[0063] Figure 4 This is a diagram illustrating a concept of model-based reinforcement learning according to an embodiment of the present invention. The MBRL method consists of step S410, in which the controller is first randomly initialized or computed using a partially known model of the system to be controlled. The controller can be randomly initialized without any knowledge of the system model. In step S420, the controller runs to collect state transition data, i.e., (x k ,u k ,x k+1 The controller can run in a scenario-based manner, where the system is first initialized to the initial state x. init The controller then runs the 'T' time step and collects state transition data. This scenario can be repeated "n" times until enough data is collected. In the next step 430, this data is used to enable the following expression --||x k+1 -f(x k ,u kThe loss function given in step 430 is minimized to learn the predictive model of the system. This function 'f' can be represented by a machine learning algorithm (e.g., a Gaussian process or a neural network). The learned model from step 430 is then passed to the next step 440 along with the cost function 460, which calculates the policy or controller for the system to perform the desired task. The cost function c(x) k ,u k This contains information about the expected tasks the system needs to perform, as a function of the state and inputs. Then, in step 450, the convergence of the obtained policy is tested, and if the convergence criterion is not yet met, the entire process is repeated.

[0064] Some embodiments of the present invention aim to estimate a statistical model of the system using collected state transition data. Several machine learning methods can be used for this step, including Gaussian processes or stochastic neural networks. A Gaussian process is a stochastic process that is entirely specified by a mean and a covariance function.

[0065] Figure 5A and Figure 5B This is a graph illustrating the estimation of random input-output relationships using a Gaussian process and the set of uncertainties that can be estimated using confidence intervals according to embodiments of the present disclosure.

[0066] A Gaussian process can be arithmetically represented using the mean function m(x) and the protocol difference function k(x,x′) of the process f(x) as follows:

[0067] m(x) = E[f(x)]

[0068] k(x,x′)=E[(f(x)-m(x))(f(x′)-m(x′))]

[0069] The Gaussian procedure is written as follows:

[0070] f(x)~N(m(x),k(x,x′)).

[0071] Please note that here x is the input to the Gaussian process, and for the specific case of estimation, the system dynamics are state-action pairs (x, x) k ,u kThe covariance function k(x,x′) is also called the kernel function. There are several effective covariance functions to choose from. Some of the most common choices are the radial basis function (RBF) kernel, the Matern kernel, the squared exponent (SE) kernel, etc. As a stochastic process, a Gaussian process is a set of random variables, and any finite set of these random variables is a multivariate Gaussian. After completing the learning process from the training dataset 511D={X,Y}, the predicted x at the test data point 512 is... * It is given by the Gaussian distribution (which can be shown using the properties of the multivariate Gaussian distribution). Mathematically, it is represented by the following relationship.

[0072] p(f(x * )|D,x * )=N(μ(x * ),Σ(x * ))

[0073]

[0074]

[0075] Where, μ(x) * ) and Σ(x * ) represents x at the test data point * Mean and variance of the Gaussian distribution. Estimation of the learning process term K,k. ** and k * Once these terms have been estimated, predictions for new test data points are obtained using the closed-form equations represented in the above equations. As a result, during the learning process, probabilistic relationships are learned for the expected input-output relationships between the current state-action pair and the next state transition.515

[0076] Some embodiments of this invention are based on the understanding that, for any nonlinear system, synthesizing a fully state-dependent strategy for the system is computationally and numerically very challenging. This requires a reliable estimation of the global model of the system, which is also a challenging problem to solve. On the other hand, computing a locally controlled trajectory is a simpler problem, but it can have poor performance because it is highly susceptible to initial conditions and is time-indexed. Furthermore, performance can be degraded if an incorrect or uncertain model exists. Therefore, to achieve good performance with a trajectory-centric controller, this invention proposes the use of a stable controller capable of ensuring high performance under bounded uncertainty.

[0077] The objective of some embodiments of this invention is to use a learned model f of the system and then design a robust trajectory-centric controller for the system. The trajectory-centric controller consists of jointly optimized open-loop state and control (x) that minimizes the cost for the system to perform the desired task. k ,u k and the locally stable strategy π θ Composition. This joint optimization is performed as an instance of a nonlinear program, which will be explained below with a brief introduction using trajectory data.

[0078] Consider discrete-time dynamic systems,

[0079] x k+1 =f(x) k ,u k )

[0080] in These are the differential state and control, respectively. Function f: Controlling the evolution of the differential state. Note that this can be achieved by using an explicit Euler integration scheme. From continuous-time systems The discrete-time formula is obtained, where Δt is the time step of the integration.

[0081] In typical applications, state and control are confined to a set. and In, that is, x k ∈X,u k ∈U. We use [K] to denote the index set {0,1,…,K}. The trajectory optimization problem is to manipulate and control u over a certain number of time steps [T-1]. k , so that the obtained trajectory {x k} k∈[T] Make the cost function c(x) k ,u k Minimize. More formally, the trajectory optimization problem aims to solve the following optimization problem.

[0082]

[0083] So that x k+1 =f(x) k ,u k )

[0084]

[0085] For k∈[T], x k ∈X,u k ∈U.

[0086] Some embodiments of this disclosure are based on the understanding that the above-described constrained trajectory optimization problem is a nonlinear optimization procedure (or NLP). Therefore, some embodiments aim to solve the nonlinear trajectory optimization procedure using the interior point method.

[0087] In the text below, we use the following abbreviations. We denote the nominal trajectory as X ≡ x0, x1, x2, x3, ..., x T-1 ,x T U≡u0,u1,u2,u3,...,u T-1 The actual trajectory followed by the system is represented as follows: We represent the local policy as π θ Where π is the policy and θ represents the policy parameters. The trajectory cost is sometimes also expressed as J = ∑ k∈[T} c(x k ,u k ).

[0088] The sample efficiency of MBRL technology makes it a very attractive option for learning optimal controllers for physical systems. Some embodiments of this invention are based on the understanding that the performance of MBRL algorithms is highly susceptible to modeling errors. Any errors in the system's predictive model can rapidly complicate matters and drive the controller into regions of high model uncertainty, making them difficult to control. This can cause the entire learning process to diverge and fail.

[0089] Some embodiments of this invention are based on the understanding that if the trajectory and the strategy optimized to achieve a locally stable trajectory are solved simultaneously, the optimizer can be constrained to avoid uncertain regions in the system's state space. This joint optimization forces the optimization to seek a trajectory that is both easily stable and optimal. Therefore, it allows us to make trade-offs between optimality and robustness in the case of an uncertain model estimated during the learning process. This results in better and more stable learning of the MBRL algorithm.

[0090] Some embodiments of this invention are based on the understanding that many trajectory stabilization techniques compute time-varying controllers. This makes real-time control of the controlled system more difficult. Generally, RL algorithms compute state-dependent strategies as outputs because they are easier to implement. The main reason is that the system is nonlinear, so even small disturbances in the system state change the system dynamics. Time-dependent tracking controllers are not state-dependent and therefore perform poorly in scenarios with disturbances in the planned trajectory. Therefore, this invention proposes to compute time-independent feedback strategies to achieve local trajectory stabilization near the nominal trajectory.

[0091] Figure 6AThis is a diagram illustrating the evolution of the stochastic dynamics of a system that can be represented by a Gaussian process. 610. Assume the system is in the initial state 613x. k And input 611u k The system dynamics are represented by a stochastic function 612f(m(x),k(x,x′)) (note that here x represents the input of the model, i.e., for (x... k ,u k )).

[0092] Consider the following case: the system dynamics f are stochastic (i.e., it is defined by mean and covariance functions), and the controller is designed using only the mean component of f. Consider the deviation of the system from the state trajectory X at any step 'k' and express it as... We introduced an adjustment for the local trajectory deviation δx k Local (time-invariant) strategy π θ Therefore, the final controller is represented as The closed-loop dynamics of the system under this control are given below:

[0093]

[0094] The main goal is to find that can Time-invariant feedback strategy π for locally stable open-loop trajectory X θ , where R k The deviation δx is defined. k The set of uncertainties. This can be addressed by using a diagonal positive definite matrix S. k Fitting the ellipsoid to the uncertainty estimate makes To approximate the uncertainty region R k The general optimization problem for achieving this is proposed as follows:

[0095]

[0096]

[0097] in, Let represent the mean function of the model. Note that in the above equation, compared to the classic trajectory optimization problem introduced earlier, we introduce a function corresponding to the policy π. θ Additional optimization parameters.

[0098] Some embodiments of this disclosure are based on the understanding that a stochastic model of the system can be used to estimate the set of system states that the system may move from the nominal trajectory during runtime. This estimation of the uncertainty set gives us the set of states that the system needs to be stabilized during runtime. The goal of the stabilization controller is to push all states in this set back to the original nominal trajectory. As long as this is true, the system is able to follow the original nominal trajectory and therefore achieve the desired optimal performance when performing the desired task.

[0099] One implementation aims to use stochastic machine learning algorithms (such as Gaussian process regression or stochastic neural networks) to estimate the uncertainty region R during the model learning process. k As previously explained, Gaussian processes can be used to learn the stochastic dynamics model of a system as a probabilistic function.

[0100] Some embodiments of the present invention use this probability relation to estimate confidence intervals using a predicted Gaussian distribution at test data points. The confidence level α∈(0,1) is a user-selected input that controls the size of the uncertainty set. For worst-case controller designs, this value is typically chosen to be close to 1.

[0101] Figure 5A and Figure 5B This indicates that at test point x * 512 The average predicted value obtained by regression through the Gaussian process is given by y(x) * 513 and 514 represent the confidence intervals. The learned Gaussian process regression provides the mean predictive estimate at any point 515 and the confidence interval at the same test point 516. Since the expected force distribution at any point is given by the Gaussian distribution, it is shown in 520 at x. * The distribution of predicted states at point 521. The mean of predicted state 521 (513 in process 510) is y(x * ), and the confidence interval is 522 (as shown in 514 of procedure 510).

[0102] also, Figure 6A It is a diagram illustrating the concept of the dynamic evolution of a system using stochastic processes, and Figure 6B This is a diagram illustrating the concept of using a stabilization strategy to stabilize a controlled trajectory during the design of a trajectory-centric controller, according to an embodiment of the present disclosure.

[0103] Some embodiments of the present invention use this uncertain region 630, estimated by machine learning methods, to simultaneously compute a stabilizing strategy with the trajectory 650. The function of the stabilizing strategy is to keep the system close to the nominal system state given by the trajectory.

[0104] The robust trajectory optimization problem is to minimize the trajectory cost while satisfying robustness constraints at each step along the trajectory. The objective 620 of the local stability controller is to push the state 640 with the worst-case deviation to the ε-tolerance sphere 620 around the trajectory at each step along the trajectory 650. Mathematically, we express this problem as follows:

[0105]

[0106] Such that for k∈[T], x k+1 =f(x) k ,u k )

[0107]

[0108] For k∈[T], x k ∈X,u k ∈U

[0109]

[0110] We define the last term in the above equation as a robustness constraint for the trajectory optimization problem. This constraint has several noteworthy characteristics. The constraint acts on the trajectory at each point along the nominal trajectory and ensures the locally stable policy π. θ Set R k The worst-case deviation of the system state is pushed toward the nominal trajectory. Furthermore, this allows users to select hyperparameters ∈ k To determine the tightness with which the nominal trajectory is followed. This hyperparameter can also be time-indexed.

[0111] Figure 7 This is a flowchart illustrating the different steps involved in solving a nonlinear optimization program used in solving a trajectory-centered controller according to an embodiment of the present disclosure. The nonlinear optimization program solver 710 accepts known (learned or fully known) system dynamics 720, the state and input set of the system 730, robustness constraints 740, and the desired task or objective function 750 as inputs. The output provided by the nonlinear optimization program solver is the optimal state input trajectory ({x... k} k∈[T] ,{u k} k∈[T] and stable strategy π θ .

[0112] Some embodiments of the present invention are based on the understanding that robustness constraints lead to sub-optimization problems within the main trajectory optimization problem. This requires estimating the stable policy π. θ The set R that produces the maximum deviation under the action of k The term δxk Then, when the deviation produced by such a point is less than the hyperparameter ∈ k At that time, robustness constraints are satisfied. Finding solutions to optimization problems with robustness constraints on arbitrary nonlinear dynamics f is very challenging. However, it is possible to solve the problem by focusing on the state control pair (x) k ,u k The problem can be simplified by linearizing the dynamics of the system, and linear dynamics can be used to find local solutions to the optimization problem within robustness constraints.

[0113] Robustness constraints for the linearized dynamic model can be obtained by using the Taylor series expansion of the nonlinear dynamic model f. Therefore, the robustness constraint term (called d) is... max The following can be simplified to the following items.

[0114]

[0115] in, and It is a model A linearized model of the mean function.

[0116] Some currently disclosed implementations are based on the understanding that robustness constraints on optimization problems may require adjustments to hyperparameters ∈ k Extensive adjustments are made to obtain a feasible solution (i.e., a solution that satisfies the constraints for all k∈[T]). In some cases, a feasible solution may not even be guaranteed. A possible solution to circumvent this possibility is to add something to the objective function to transform the hard robustness constraints into soft constraints, thereby penalizing them on each violation. In the simplest setting, the robustness term d max,k It is possible to simply move to the objective function and minimize it during optimization. This relaxes the constraints on robustness while ensuring the feasibility of the optimization problem.

[0117] Some embodiments of the present invention can use robustness constraints as soft constraints and move them to the objective function to solve the following optimization problem.

[0118]

[0119] Such that for k∈[T], x k+1 =f(x) k ,u k )

[0120]

[0121] For k∈[T], x k ∈X,u k ∈U

[0122] Some embodiments of this disclosure are based on the understanding that, in order to solve nonlinear trajectory optimization problems with robust constraints (either hard or soft), it is necessary to compute the gradients of the robust constraints. However, since the maximum function in the constraints is nondifferentiable, the robust constraints are also nondifferentiable. However, the first-order gradient of the robust term can be obtained using Danskin's theorem. This implementation is crucial for solving trajectory optimization problems. Therefore, the aim of some embodiments is to estimate the gradients of the robust constraints using Danskin's theorem.

[0123] More formally, set Let be a non-empty, closed set, and assume... Let g be a non-empty, open set. Assume the function g: Ω×K→R is continuous on Ω×K, and... It exists and is continuous over Ω×K. The function h:Ω→R∪{∞} is defined as follows:

[0124]

[0125] as well as

[0126] M(x)≡{y∈K|h(x)=g(x,y)}.

[0127] Let x∈Ω be a given vector. Assume the neighborhood of x There exists such that for all x′∈N(x), M(x′) is nonempty and the set ∪ x′∈N(x) M(x′) is bounded. Then, according to Danskin's theorem, the following two statements are valid.

[0128] The function h is differentiable in a direction with respect to x, and

[0129]

[0130] If M(x) simplifies to a single element, for example, M(x) = {y(x)}, then h is Gaeaux differentiable with respect to x, and

[0131]

[0132] Dankin's theorem allows us to find the gradient of robustness constraints by first calculating the parameters of the maximum function and then evaluating the gradient of the maximum function at that point.

[0133] Therefore, in order to find the gradient of robust constraints, it is necessary to interpret it as a gradient in δx kThe optimization problem in this context will be discussed later. Some implementations of this disclosure are based on the understanding that finding the gradients of robustness constraints in general form can be very challenging. A simplification of robustness constraints is achieved by using linear parameterization of the stable policy, which allows the parameters of the maximum function in the robustness constraints to be computed analytically. This is highly advantageous for computational requirements, as it can be obtained analytically without any iterative optimization process.

[0134] In mathematics, we represent a stable strategy as a time-invariant constant matrix W, such that π θ (δx k )=W·δx k The following implementation then guides us to estimate the parameters of the maximum function under robustness constraints.

[0135] Robustness constraint term d for linear parameterization of the feedback strategy max,k In parameter δx k The denoted is quadratic. Mathematically, it is written as follows:

[0136]

[0137] So that, Among them, M k (x k ,u k ,W)≡A(x k ,u k )+B(x k ,u k )·W.

[0138] Worst-case deviation term d max,k The resulting form allows us to compute it as the largest eigenvalue λ of matrix Q through analysis. max , where the matrix

[0139] Worst deviation δx max The eigenvector corresponding to the largest eigenvalue.

[0140] Then Dankin's theorem can be used to obtain the robustness constraint d. max,k The gradient is used as δx max The gradient of the term. More specifically, using the sign of an arbitrary vector z, and using... robustness constraint term d max The gradient of any vector z can be calculated as follows:

[0141]

[0142] This provides us with the gradient of the robustness constraint term over time step "k". Some implementations aim to compute the gradient of the robustness constraint term for all k∈[T].

[0143] Figure 8 This is a flowchart illustrating the different steps of estimating gradients for robust constraints in trajectory-centric controller synthesis according to embodiments of the present disclosure. Around the nominal state, action pair (x... k ,u k The estimation of the uncertain region is calculated in step 810, and then in step 820, the nominal state, action pair (x) is used. k ,u k The worst-case deviation state in the uncertain region is calculated. Then, in step 830, this worst-case deviation state is used by another implementation to calculate the gradient of the robustness constraint. Then, in step 840, the gradient information is transmitted to the nonlinear optimization program solver.

[0144] Then, the gradients of robustness constraints and the dynamic gradients of the system can be used to solve the robust policy optimization problem to obtain the optimal solution. Some embodiments of this invention use the cost function of the trajectory, the dynamics of the system, robustness constraints using the uncertain regions of the system, and gradient information of the dynamics to solve a robust nonlinear optimization procedure, thereby simultaneously synthesizing the open-loop trajectory X = x1, x2, ..., x T U = u1, u2, ..., u T-1 and local stable strategy π θ (δx k )=W·δx k .

[0145] Simulation tests were performed using nonlinear systems to test and verify the benefits of the proposed robust, goal-centered controller. Specifically, can the feedback controller obtained through this optimization stabilize the open-loop trajectory in the face of bounded uncertainties? This problem is investigated using an underactuated pendulum system. The aim is to demonstrate the effectiveness of the solution in the stable optimal trajectory and to compare the feedback gain obtained through optimization with a time-varying LQR solution. The dynamics of the pendulum are modeled as follows: The continuous-time model is discretized into The target state is x g = [π, 0], and the initial state is x0 = [0, 0].

[0146] Figure 9An illustrative example of the effectiveness of the proposed controller on an inverted pendulum system under unknown dynamics is shown. Because the system dynamics are incorrect, the open-loop controller 920 cannot achieve the target, but the stable controller 910 can stabilize the entire trajectory using a time-invariant gain. Figure 10A The feedforward (open-loop) control trajectory is shown. Figure 10B The time-invariant feedback matrix π is shown. θ =Gain of W (Note that there are two gains, one for the angular position θ of the pendulum and the other for the angular velocity) ). Figure 10C The time-varying LQR gain obtained by solving the LQR problem at each point of the trajectory using linearized dynamics A(k) and B(k) is shown. Figure 11 The statistical behavior of the proposed controller for the same system under several different disturbances injected into the system is shown. As can be seen, the controller always tends to 0.

[0147] The proposed robust trajectory-centered controller is tested on a real ball-beam system with unknown dynamics. The ball-beam system is a low-dimensional nonlinear system exhibiting nonlinearity due to dry friction and delay in the actuator. Four state variables are used. Model the sphere-beam system, where x is the position of the sphere. θ is the velocity of the ball, and θ is the angular position of the beam. That is the angular velocity of the beam. The acceleration of the ball is given by the following nonlinear equation:

[0148]

[0149] Where, m ball It is the mass of the ball, I ball It is the moment of inertia of the ball, r ball Let b1 be the radius of the ball, b2 be the coefficient of viscous friction of the ball on the beam, and b3 be the coefficient of dry friction of the ball on the beam. An inaccurate model of the system is used to design an open-loop trajectory and a robust trajectory-centric controller. Even with an inaccurate model, the proposed robust controller achieves better performance than the open-loop controller.

[0150] Figure 12 A ball-beam system is shown, consisting of a ball 1150 that can move on a beam 1120. The initial position of the ball is fixed at 1110. The task is to move the ball to the desired position and hold it there. The inclination angle of the beam is measured by an encoder 1130 and an encoder shaft 1140. Figure 13 A typical behavior example of the proposed trajectory-centric controller compared to an open-loop controller is shown. Figure 13The average steady-state error 1210 of the proposed controller is shown to approach zero. However, the open-loop controller performs poorly with a non-zero steady-state error 1220.

[0151] The embodiments described above in this disclosure can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can execute on any suitable processor or set of processors, whether located in a single computer or distributed across multiple computers. Such a processor can be implemented as an integrated circuit having one or more processors within an integrated circuit assembly. However, the processor can be implemented using circuitry of any suitable format.

[0152] Furthermore, the various methods or processes outlined herein can be encoded as software executable on one or more processors, which employ any of a variety of operating systems or platforms. Additionally, such software can be written using any of a large number of suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine. Typically, the functionality of program modules can be combined or distributed as needed in various implementations.

[0153] Furthermore, embodiments of this disclosure can be embodied as a method, examples of which have been provided. Actions performed as part of this method can be ordered in any suitable manner. Therefore, embodiments can be constructed in which actions are performed in a different order than those shown, which may include performing several actions simultaneously, even if these actions are shown as sequential in the exemplary embodiments. Moreover, the use of ordinal terms such as "first" or "second" to modify claim elements in the claims does not in itself imply any priority, precedence, or order of one claim element relative to another claim element, or the temporal order of performing method actions, but merely serves as a label to distinguish one claim element with a certain name from another element with the same name (except for the use of ordinal numbers), thus differentiating claim elements.

[0154] Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications can be made within the spirit and scope of this disclosure. Therefore, the appended claims cover all such variations and modifications that fall within the true spirit and scope of this disclosure.

Claims

1. A controller for optimizing a local control policy for a system using trajectory-centric reinforcement learning, the controller comprising: An interface configured to receive data including tuples of system state, control data, and state transitions measured by sensors; The memory stores a program that the processor can execute, the program including a stochastic predictive learning model for generating a nominal state and control trajectory as a function of time steps within a desired time range in response to a task command received from the system via the interface; a control strategy including a machine learning method algorithm and an initial stochastic control strategy; and the local control strategy for adjusting deviations along the nominal trajectory, the control trajectory being an open-loop trajectory. At least one processor, said at least one processor being configured to: The system's stochastic prediction model is learned using a set of data collected during trial-and-error experiments performed using the initial stochastic control strategy. Estimate the average forecast and uncertainty associated with the stochastic forecast model; The trajectory-centric controller synthesis problem is formulated to simultaneously compute the nominal trajectory, feedforward control, and steady-state time-invariant feedback control. When using control inputs at the current time step, a local set of deviations from the nominal system state is determined using a learned stochastic system model; Identify the system state that has the worst-case deviation from the nominal system state from the local set of system deviations; The gradient of the robustness constraint is determined by calculating the first derivative of the robustness constraint under the system state with worst-case bias. By using nonlinear programming to solve robust strategies to optimize and minimize the cost of the state control trajectory while satisfying the state and input constraints that serve as the local control strategy, the optimal system state trajectory, feedforward control input, and local time-invariant feedback strategy for adjusting the system state to the nominal trajectory are determined. The control data is updated based on the solved optimization problem; and The updated control data is output via the interface.

2. The controller according to claim 1, wherein, The system is a discrete-time dynamic system.

3. The controller according to claim 1, wherein, The at least one processor is configured to perform the trajectory-centered reinforcement learning of the system by synthesizing a trajectory-centered control strategy through time-dependent feedforward control and local time-invariant feedback control that stabilizes the time-dependent feedforward control.

4. The controller according to claim 2, wherein, The synthesis of trajectory-centric control strategies for discrete-time dynamic systems is formulated as a nonlinear optimization procedure with nonlinear constraints.

5. The controller according to claim 4, wherein, The nonlinear constraints are system dynamic and stability constraints used for local time-invariant feedback strategies.

6. The controller according to claim 1, wherein, The local time-invariant feedback strategy is determined as a solution to the mathematical expression of a robust trajectory optimization problem, which minimizes the trajectory cost and additionally satisfies the robustness constraints at each step along the trajectory. These robustness constraints push the current state of the system, which is in a worst-case deviation state at the current time step along the trajectory, into a tolerance range around the trajectory at the next time step.

7. The controller according to claim 1, wherein, The set of local uncertainties along the nominal trajectory is obtained by a stochastic function approximator used to learn the forward dynamics model of the system.

8. The controller according to claim 1, wherein, The worst-case deviation state of the system at each state along the nominal trajectory in the defined set of system states is obtained by solving an optimization problem.

9. The controller according to claim 1, wherein, Using the gradient of the robustness constraint at the worst-case deviation state, solve the formulaic nonlinear procedure with robustness constraints to obtain the feedforward control and the time-invariant feedback control, or Wherein, at least one of the sensors performs wireless communication via the interface, or At least one of the sensors is a 3D camera that provides motion pictures including depth images, or The trajectory-centered controller synthesis problem is a nonlinear procedure, or The local strategy is either a time-invariant feedback strategy or a locally stable controller, or The control trajectory is an open-loop trajectory.

10. The controller according to claim 9, wherein, The sensor is arranged in the system and in a predetermined peripheral location.

11. The controller according to claim 10, wherein, At least one of the predetermined peripheral positions is determined by a viewing angle, such that the 3D camera captures the range of motion of the system.

12. A computer implementation method for optimizing a local control policy for a trajectory-centered reinforcement learning system, the method comprising the following steps: The system's stochastic prediction model is learned using a set of data collected during trial-and-error experiments performed using an initial stochastic control strategy. Estimate the average forecast and uncertainty associated with the stochastic forecast model; The trajectory-centered controller synthesis problem is formulated to simultaneously compute the nominal trajectory, feedforward control, and steady-state time-invariant feedback control. When using control inputs at the current time step, a local set of deviations from the nominal system state is determined using a learned stochastic system model; Identify the system state that has the worst-case deviation from the nominal system state from the local set of system deviations; The gradient of the robustness constraint is determined by calculating the first derivative of the robustness constraint under the system state with worst-case deviation. By using nonlinear programming to provide and solve a robust policy optimization problem that satisfies the state and input constraints that serve as the local control policy while minimizing the cost of the state control trajectory, the optimal system state trajectory, feedforward control input, and local time-invariant feedback policy for adjusting the system state to the nominal trajectory are determined. Update the control data based on the solved optimization problem; as well as The updated control data is output via the interface.

13. The method according to claim 12, wherein, The system is a discrete-time dynamic system.

14. The method according to claim 12, wherein, In order to perform the trajectory-centered reinforcement learning of the system, a trajectory-centered control strategy is synthesized by time-dependent feedforward control and local time-invariant feedback control that stabilizes the time-dependent feedforward control.

15. The method according to claim 13, wherein, The synthesis of the trajectory-centered control strategy for the discrete-time dynamic system is formulated as a nonlinear optimization procedure with nonlinear constraints.

Images