Method and system for modeling and controlling a partially measurable system

Abstract

A controller is provided for a control system, the system including a strategy configured to control the system. The controller includes: an interface connected to the system, the interface configured to acquire action states and measurement states via sensors of the measurement system; a memory for storing a computer-executable program module including a model learning module and a policy learning module; and a processor configured to execute the steps of the program module. The steps include: offline modeling to generate offline learned states based on the action states and measurement states using the model learning program; providing the offline states to a policy learning program to generate policy parameters; and updating the system's policy based on the policy parameters to operate the system. In the policy learning program for generating policy parameters, a dropout method for improving policy parameter optimization, a particle method for calculating and evaluating the evolution of particle states, and a sensor model and an online estimator model for generating online estimates of particle states are taken into account to approximate state estimates based on particle states generated from the model learning program.

Description

Technical Field

[0001] The present invention generally relates to methods and systems for modeling and controlling partial and fully measurable systems, including mechanical systems. Background Technology

[0002] In recent years, reinforcement learning (RL) has achieved remarkable results in many different environments and has shown the potential to provide automation frameworks for learning from scratch for various control applications. However, model-free RL (MFRL) algorithms may require extensive interaction with the environment to solve the assigned task. Data inefficiency, due to the time and cost of interacting with the environment, limits the potential of RL in real-world applications. In particular, when dealing with mechanical systems, it is crucial to learn the task after the minimum possible amount of trial and error to reduce wear and tear and avoid any damage to the system.

[0003] There is a need to develop a method that considers modeling and filtering different components before model learning and policy optimization. Summary of the Invention

[0004] The objective of some embodiments of the present invention is to provide a promising approach to overcome the aforementioned limitations, namely, Model-Based Reinforcement Learning (MBRL), which builds a predictive model of the environment using data from interactions and leverages it to plan control actions. MBRL improves data efficiency by using models to extract more valuable information from available data.

[0005] Some embodiments of the present invention are based on the understanding that MBRL methods are effective only when their models accurately resemble real systems. Therefore, deterministic models can be greatly affected by model inaccuracies, and the use of stochastic models becomes necessary to capture uncertainty. Gaussian processes (GPs) are a class of Bayesian models commonly used precisely in RL methods due to their inherent ability to handle uncertainty and provide principled stochastic predictions. Furthermore, PILCO (Probabilistic Inference Learning Control) can be a successful MBRL algorithm, using GP models and gradient-based policy search to achieve substantial data efficiency in solving various control problems in both simulated and real systems. In PILCO, long-term predictions are computed analytically, approximating the distribution of the next state at each time step using a Gaussian distribution through moment matching. In this way, the policy gradient is computed in a closed-form. However, the use of moment matching can introduce two related challenges. (i) Moment matching allows modeling only unimodal distributions. This fact, besides introducing potentially incorrect assumptions about system dynamics, also introduces relevant constraints related to initial conditions. In particular, the restriction on the use of unimodal distributions complicates the handling of multimodal initial conditions and is a potential limitation even when the initial state of the system is unimodal. For example, with high initial variance, the optimal solution may be multimodal due to its dependence on initial conditions. (ii) The computation of moments is only proven tractable when considering the squared exponent (SE) kernel and a differentiable cost function. In particular, the restrictions on kernel selection can be very strict, as the GP with the SE kernel imposes a smoothing property on the posterior estimator and may exhibit poor generalization properties on data not seen during training.

[0006] Furthermore, some embodiments of the present invention are based on the understanding that PILCO has inspired several other MBRL algorithms, which attempt to improve upon it in different ways. Limitations arising from the use of SE kernels have been addressed in Deep-PILCO, where Bayesian neural networks can be used to model system evolution, and particle-based methods and moment matching are combined to compute long-term predictions. Results show that Deep-PILCO requires significantly more interaction with the system to learn the task compared to PILCO. This fact suggests that using neural networks (NNs) may not be advantageous in terms of data efficiency due to the considerable number of parameters required to characterize the model. A more explicit approach is to use a set of probabilities from NNs to model the uncertainty of system dynamics. While results in simulated high-dimensional systems are positive, numerical results show that GPs are more data-efficient than NNs when considering low-dimensional systems such as a cart-turned-inverted-pendulum reference. An alternative approach is to use a simulator to learn the prior of the GP model before initiating the reinforcement learning process on a real system for control. This simulated prior can improve PILCO's performance in regions of the state space where no data points are available. However, this method requires an accurate simulator that may not always be available to the user. Some challenges arise from addressing gradient-based optimization in Black-DROPS, which employs a gradient-free optimization strategy. Some implementations are based on the understanding that a non-differentiable cost function can be used, and that parallelization of the black-box optimizer can be leveraged to improve computation time. Using this strategy, Black-DROPS achieves data efficiency similar to PILCO, but with significantly improved asymptotic performance.

[0007] Furthermore, some embodiments of the present invention are based on the understanding that other methods exist focused on improving the accuracy of long-term predictions and overcoming approximation problems caused by moment matching. One attempt could be a method relying on particle-based approaches to compute long-term distributions. Based on the current policy and a one-step-ahead GP model, the evolution of a batch of particles sampled from an initial state distribution can be simulated. The expected cumulative cost is then approximated using particle trajectories. The policy gradient can be computed using a policy, where a probabilistic Markov decision process (MDP) is transformed into an equivalent partially observable MDP with deterministic transitions by fixing the initial random seed. The results obtained are unsatisfactory compared to PILCO. The poor performance is attributed to the policy optimization method, particularly its inability to escape multiple local minima generated by a multimodal distribution. Another particle-based approach could be PIPPS, where the policy gradient is computed using a so-called reparameterization trick instead of a PEGASUS policy. Given a distribution p... θ (·), parameterized using θ, the reparameterization technique provides a way to obtain p θ(·) An alternative method for generating samples, such that these samples are differentiable with respect to θ. The reparameterization trick has been successfully introduced into Stochastic Variational Inference (SVI). In contrast to the results obtained in SVI, where only a few samples are needed to estimate the gradient, several problems may exist related to the gradient computed using the reparameterization trick due to its exploding magnitude and random direction. To overcome these problems, a total propagation algorithm is proposed, in which the reparameterization trick is combined with the likelihood ratio gradient. This algorithm is performed similarly to PILCO, with some improvements in gradient computation and performance with additional noise.

[0008] Some implementations disclose an MBRL algorithm called Monte Carlo Probabilistic Learning Control Inference (MC-PILCO). Similar to PILCO, MC-PILCO is a policy gradient algorithm that uses a generalized approach (GP) to describe the system dynamics one step ahead and relies on a particle-based approach to approximate the long-term state distribution instead of moment matching. The gradient of the expected cumulative cost with respect to the policy parameters is obtained by backpropagation on an associated stochastic computation graph, utilizing reparameterization techniques. Unlike PIPPS, which focuses on obtaining an accurate estimate of the gradient, the optimization problem can be interpreted as a stochastic gradient descent (SGD) problem. This problem has been extensively studied in the context of neural networks, where noisy estimates of the gradient are used to optimize the overparameterized model. Analytical and experimental studies show that the shape of the cost function and nonlinear activation function employed can significantly affect the performance of the SGD algorithm. Inspired by results related to previous particle-based methods in the art, we consider using more complex policies and smoother cost functions (i.e., less cost penalty). During policy optimization, we also consider applying dropout techniques to the policy parameters to improve the ability to escape local minima, thereby obtaining a higher-performance policy. The effectiveness of the proposed choice was analyzed and validated in simulations. First, considering the simulated inverted pendulum cart as a common benchmark system, MC-PILCO was compared with PILCO and Black-DROPS. The results show that MC-PILCO outperforms PILCO and Black-DROPS, the latter two of which can be considered state-of-the-art algorithms for GP-based MBRL. Second, to evaluate the behavior of MC-PILCO in higher-dimensional systems, we applied it to a simulated UR5 robot arm. The task considered involved learning a joint space controller capable of tracking a desired trajectory, which was successfully completed. These results confirm that reparameterization techniques can be effectively used in MBRL, and that the Monte Carlo method, when properly accounted for the cost function, dropout techniques, and the use of complex / rich policies, does not encounter gradient estimation problems, which differs from what is commonly stated in the literature.

[0009] Furthermore, unlike previous work combining GP with particle-based methods, we demonstrate a significant advantage of this strategy: the possibility of employing different kernel functions. We consider using kernel functions composed of combinations of SE kernels and polynomial kernels, along with semi-parametric models. Results obtained in simulated and real Furuta pendulum experiments show that using such kernels significantly improves data efficiency and limits the interaction time required for the learning task.

[0010] Finally, MC-PILCO is applied and analyzed in partially measurable systems, and is referred to as MC-PILCO4PMS. Unlike simulated environments where the state is typically assumed to be fully measurable, the state of a real system may be partially measurable. For example, in most real robotic systems, only position is directly measurable, while velocity is usually calculated using estimators such as state observers, Kalman filters, and numerical differentiation with low-pass filters. In particular, the controller (i.e., the policy) operates using the output of an online state estimator, which can introduce significant latency and inconsistencies related to the filtered data used during policy training due to noise and real-time computational constraints. In this case, we validate that during policy optimization, it is important to distinguish between the state generated by the model (which is intended to describe the evolution of the real system state) and the state provided to the policy. In fact, providing model predictions to the control policy is equivalent to assuming direct measurement of the system state, as mentioned earlier, which is impossible in real systems. This flawed assumption can impair the effectiveness of training the policy for use in real systems due to the presence of distortions introduced by the online state estimator. Therefore, during policy optimization, we compute estimates of the observed states by modeling both the measurement system and the online estimator used in the real system, based on the evolution of the system states predicted by the GP model. We then provide these estimates of the observed states to the policy. In this way, we aim to obtain robustness to delays and distortions caused by online filtering. The effectiveness of the proposed policy has been tested in simulations and two real systems (Furuta pendulum and ball-and-disc systems). The obtained performance confirms the importance of considering the presence of filters in the real system during policy optimization.

[0011] Some embodiments of the present invention are based on the understanding that a controller for a control system can be provided, the system including a strategy configured to control the system. In this case, the controller may include: an interface connected to the system, the interface being configured to acquire action states and measurement states via sensors measuring the system; a memory for storing a computer-executable program module including a model learning module and a strategy learning module; and a processor configured to execute the steps of the program module. Furthermore, the steps include: offline modeling to generate an offline learning state based on the action state and measurement state using the model learning program, wherein the model learning module includes an offline state estimator and a model learning module, wherein the offline state estimator estimates the offline state and provides it to the model learning program, wherein the policy learning module includes a system model, a sensor model, an online state estimator model, and a policy optimization program, wherein the system model generates particle states that approximate the real system state, wherein the sensor model approximates particle measurements, and the particle measurements approximate measurements on the real system based on the particle states, wherein the online state estimator model is configured to generate an online particle estimate based on the particle measurements and possible previous online particle estimates, wherein the policy optimization program generates policy parameters; providing the offline state to the policy learning module to generate policy parameters; and updating the system's policy based on the policy parameters to operate the system.

[0012] According to another embodiment of the present invention, a vehicle control system for controlling the motion of a vehicle is provided. The vehicle control system may include a controller, the controller may include an interface connected to the system configured to acquire motion states and measurement states via sensors of a measurement system; a memory for storing a computer-executable program module including a model learning module and a policy learning module; and a processor configured to execute the steps of the program module. Furthermore, the steps include: offline modeling to generate an offline learning state based on the action state and measurement state using the model learning program, wherein the model learning module includes an offline state estimator and a model learning module, wherein the offline state estimator estimates the offline state and provides it to the model learning program, wherein the policy learning module includes a system model, a sensor model, an online state estimator model, and a policy optimization program, wherein the system model generates particle states that approximate the real system state, wherein the sensor model approximates particle measurements, the particle measurements approximate measurements on the real system based on the particle states, wherein the online state estimator model is configured to generate an online particle estimate based on the particle measurements and possible previous online particle estimates, wherein the policy optimization program generates policy parameters; providing the offline state to the policy learning module to generate policy parameters; and updating the system's policy based on the policy parameters to operate the system, wherein the controller is connected to the vehicle's motion controller and a vehicle motion sensor that measures the vehicle's motion, wherein the control system generates policy parameters based on the motion measurement data, and wherein the control system provides the policy parameters to the vehicle's motion controller to update the motion controller's policy unit.

[0013] Furthermore, some embodiments of the present invention provide a robot control system for controlling the movement of a robot. This robot control system may include: an interface connected to the system, configured to acquire motion and measurement states via sensors of a measurement system; a memory for storing a computer-executable program module including a model learning module and a policy learning module; and a processor configured to execute the steps of the program module. Furthermore, the steps include: offline modeling to generate an offline learning state based on the action state and measurement state using the model learning program, wherein the model learning module includes an offline state estimator and a model learning module, wherein the offline state estimator estimates the offline state and provides the offline state to the model learning program, wherein the policy learning module includes a system model, a sensor model, an online state estimator model, and a policy optimization program, wherein the system model generates particle states that approximate the real system state, wherein the sensor model approximates particle measurements, the particle measurements approximate measurements on the real system based on the particle states, wherein the online state estimator model is configured to generate an online particle estimate based on the particle measurements and possible previous online particle estimates, wherein the policy optimization program generates policy parameters; providing the offline state to the policy learning module to generate policy parameters; and updating the system's policy based on the policy parameters to operate the system, wherein the controller is connected to the robot's actuator controller and sensors configured to measure the robot's state, wherein the control system generates policy parameters based on the sensor measurement data, and wherein the control system provides the policy parameters to the robot's actuator controller to update the actuator controller's policy unit. Attached Figure Description

[0014] The accompanying drawings, included to provide a further understanding of the invention, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0015] [ Figure 1A ] Figure 1A A block diagram is shown illustrating a system controlled via a controller according to an embodiment of the present invention;

[0016] [ Figure 1B ] Figure 1B A block diagram is shown, illustrating a controller configured to control a vehicle's suspension system according to an embodiment of the present invention;

[0017] [ Figure 1C ] Figure 1C A block diagram is shown illustrating a controller for an actuator system configured to control a robot according to an embodiment of the present invention;

[0018] [ Figure 1D ] Figure 1D A schematic diagram is shown, illustrating a particle-based strategy optimization method for a control system according to an embodiment of the present invention;

[0019] [ Figure 1E ] Figure 1E A table showing an embodiment of the standard values ​​of the strategy optimization parameters according to an embodiment of the present invention is provided;

[0020] [ Figure 1F ] Figure 1F An algorithm for MC-PILCO according to an embodiment of the present invention is shown;

[0021] [ Figure 2A ] Figure 2A The graphs show the performance of the MC-PILCO algorithm according to an embodiment of the present invention under two different configurations of the parameters of the cost function;

[0022] [ Figure 2B ] Figure 2B A table showing the percentage success rate of the MC-PILCO algorithm according to an embodiment of the present invention under two different configurations of the parameters of the cost function is presented;

[0023] [ Figure 3A ] Figure 3A The graphs showing the performance of the MC-PILCO algorithm according to an embodiment of the present invention, with and without considering the discarding technique, are illustrated.

[0024] [ Figure 3B ] Figure 3B A table showing the percentage success rate of the MC-PILCO algorithm according to an embodiment of the present invention, with and without considering the drop-out method;

[0025] [ Figure 4A ] Figure 4A A graph showing the performance of the MC-PILCO algorithm according to an embodiment of the present invention when using two different kernel functions is illustrated.

[0026] [ Figure 4B ] Figure 4B A table showing the percentage success rate of the MC-PILCO algorithm when using two different kernel functions according to an embodiment of the present invention is provided.

[0027] [ Figure 5A ] Figure 5A The graph shows the performance of the MC-PILCO algorithm according to an embodiment of the present invention when using a full-state model or a velocity integral model during model learning;

[0028] [ Figure 5B ] Figure 5BA table showing the percentage success rate of the MC-PILCO algorithm when using a full-state model or a velocity integral model during model learning according to an embodiment of the present invention is provided.

[0029] [ Figure 6A ] Figure 6A A graph comparing the performance of algorithms MC-PILCO, Black-DROPS, and PILCO according to embodiments of the present invention is shown.

[0030] [ Figure 6B ] Figure 6B A comparison table showing the success rate percentages of the algorithms MC-PILCO, Black-DROPS, and PILCO according to embodiments of the present invention is provided.

[0031] [ Figure 6C ] Figure 6C A comparison table of errors for algorithms MC-PILCO (rows 1-5), Black-DROPS (row 6), and PILCO (row 7) with different options according to embodiments of the present invention is shown.

[0032] [ Figure 7A ] Figure 7A A graph showing the behavior of a cart-based inverted pendulum system controlled by the MC-PILCO algorithm according to an embodiment of the present invention is shown.

[0033] [ Figure 7B ] Figure 7B A graph showing the behavior of a cart-mounted inverted pendulum system controlled by the PILCO algorithm according to an embodiment of the present invention is shown.

[0034] [ Figure 8 ] Figure 8 A block diagram of a control scheme for trajectory tracking of a robot manipulator according to an embodiment of the present invention is shown;

[0035] [ Figure 9A ] Figure 9A A graph illustrating the trajectory of a robot manipulator controlled by the MC-PILCO algorithm according to an embodiment of the present invention, showing how to learn to track a circular trajectory;

[0036] [ Figure 9B ] Figure 9B A table showing the average and maximum errors of a robot manipulator controlled by the MC-PILCO algorithm according to an embodiment of the present invention for learning how to track a circular trajectory is presented.

[0037] [ Figure 10A ] Figure 10AA block diagram representing particle generation for strategy optimization of the MC-PILCO algorithm is shown according to an embodiment of the present invention.

[0038] [ Figure 10B ] Figure 10B A block diagram representing particle generation for strategy optimization of the algorithm MC-PILCO4PMS is shown according to an embodiment of the present invention.

[0039] [ Figure 11A ] Figure 11A Two graphs illustrating the operation of the inverted pendulum system are shown. According to an embodiment of the invention, in the top graph, the system is controlled by the MC-PILCO algorithm, and in the bottom graph, the system is controlled by the MC-PILCO4PMS algorithm.

[0040] [ Figure 11B ] Figure 11B Two graphs illustrating the evolution of the inverted pendulum system over test time are shown. According to an embodiment of the invention, in the top graph, the system is controlled by the MC-PILCO algorithm, and in the bottom graph, the system is controlled by the MC-PILCO4PMS algorithm.

[0041] [ Figure 12A ] Figure 12A A photograph of a real Furuta pendulum system according to an embodiment of the present invention is shown;

[0042] [ Figure 12B ] Figure 12B A photograph of a real ball-and-dish system according to an embodiment of the present invention is shown;

[0043] [ Figure 13A ] Figure 13A A graph showing the trajectory of the vertical angle of the Furuta pendulum system as the algorithm MC-PILCO4PMS is learning how to control the system according to an embodiment of the present invention is shown.

[0044] [ Figure 13B ] Figure 13B The diagram illustrates the trajectory of the horizontal angle of the Furuta pendulum system as the algorithm MC-PILCO4PMS learns how to control the system according to an embodiment of the present invention; and

[0045] [ Figure 14 ] Figure 14 The diagram shows the trajectory of a ball on a ball-disc system controlled by the algorithm MC-PILCO4PMS according to an embodiment of the present invention, starting from several different initial conditions. Detailed Implementation

[0046] Various embodiments of the invention are described below with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and elements with similar structures or functions are indicated by similar reference numerals in all the drawings. It should also be noted that the drawings are intended only to facilitate the description of specific embodiments of the invention. They are not to be considered an exhaustive description of the invention or a limitation on its scope. Furthermore, aspects described in connection with a particular embodiment of the invention are not necessarily limited to that embodiment, but can be practiced in any other embodiment of the invention.

[0047] According to some embodiments of the invention, there is an advantage in providing higher performance in partially measurable systems due to the consideration of two different state estimators. Partially measurable systems are systems in which only a subset of state components can be directly measured, and the remaining components can be estimated by a suitable state estimator. Partially measurable systems are particularly relevant in real-world applications because they include, for example, mechanical systems (e.g., vehicle and robotic systems) where typically only position is measured, while velocity is estimated via numerical differentiation or more complex filters. Some embodiments of the invention model the presence of two different state estimators, wherein the model learning module is an offline state estimator, and wherein the policy learning module is a sensor model and an online state estimator model. According to some embodiments of the invention, the presence of the offline state estimator improves the accuracy of the model learning module, and the presence of the sensor model and the online state estimator model improves the performance of the policy learning module.

[0048] Figure 1A A block diagram is shown illustrating a system module 10 controlled by a controller 100 according to an embodiment of the present invention. The components of the system module 10 are shown in the block diagram, which illustrates general embodiments of the controller 100 to which some embodiments of the present invention may be applied. The controller 100 represents a block diagram of some embodiments of the present invention.

[0049] In system module 10, components 11, 12, 13, and 14 represent schematic diagrams of policy execution and data collection in the real system 11. Component 11 represents a real physical system 11 that can be controlled by a controller 100 according to some embodiments of the present invention. Embodiments of component 11 may be a vehicle system, a robot system, and a suspension system. The real system 11 may be controlled by a control signal u (action state). u Due to the influence of [the system], the control signal causes the real system 11 to move into a certain state. This state is then measured by sensor 12. The state definition changes according to the real system 11.

[0050] When the real system 11 is a vehicle, the state can be the vehicle's orientation, steering angle, and velocities along the two axes of the vehicle. When the real system 11 is a robot system, the state can be joint positions and joint velocities. When the real system 11 is the vehicle's suspension system, the state can be the displacement and velocity of the suspension system from its rest position. Sensor 12 measures the state and outputs the measured values ​​of the state. Most common sensors can measure only a portion of the state, not all components of the state. For example, positioning sensors such as encoders, potentiometers, proximity sensors, and cameras can measure only the position component of the state; other sensors such as tachometers, laser surface velocimeters, and piezoelectric sensors can measure only the velocity component of the state; other sensors such as accelerometers can measure only the acceleration of the system. None of these sensors output the complete state; therefore, the measured values ​​(the output of sensor 12) are only a portion of the state. Therefore, the controller 100 according to some embodiments of the invention can control the real system 11 based on processing a portion of the measurable state of the system. An online state estimator 13 takes the measured values ​​as input and estimates the state, called state estimation, attempting to approximate the portion of the state that is not present in the measured values. Strategy 14 is a controller parameterized by means of some strategy parameters. Strategy 14 takes the state estimate as input and outputs a control signal to control the real system 11. Implementations of the strategy may include Gaussian processes, neural networks, deep neural networks, proportional-integral-derivative (PID) controllers, etc.

[0051] The controller 100 includes an interface controller (hardware circuitry), a processor, and a memory unit. The processor may be one or more processor units, and the memory unit may be a memory device, a data storage device, etc. The interface controller may be an interface circuit, which may include analog-to-digital (A / D) and digital-to-analog (D / A) converters for signal / data communication with the sensors 12 arranged in system module 10. Furthermore, the interface controller may include memory to store data to be used by the A / D or D / A converters. The sensors 12 are arranged in system module 10 to measure the state of the real system 11.

[0052] The controller 100, comprising the model learning module 1300 and the policy learning module 1400, represents some embodiments of the present invention. During policy execution in system module 10, measurements and control signals are collected as data. This data is processed by an offline state estimator 131, which filters the data and outputs an offline state estimate. The offline state estimate approximates the inaccessible state of the real system 11 because the sensor 12 only outputs measurements. Embodiments of the offline state estimator 131 may include a noncausal filter, a Kalman smoother, a central difference velocity approximator, etc. The model learning module (program module) 132 takes the offline state estimate as input and learns a system model simulating the real system. The model learning module 132 may be a Gaussian process, a neural network, a physical model, or any machine learning model. The output of the model learning module 132 is a system model 141 that approximates the real system 11.

[0053] In the strategy learning module 1400, a strategy for controlling the system is learned. Components 141, 142, and 143 of the strategy learning module 1400 approximate components of the system module 10, the real system 11, the sensor 12, and the online state estimator 13, respectively. Component 141 can be a system model 141 configured to approximate the real system 11, sensor model 142 approximates the sensor 12, component 143 approximates the online state estimator 13, and strategy optimization 144 is configured to optimize the strategy parameters defining the strategy block 14 in the system module 10. System model 141 is configured to approximate the real system 11, and when a control signal is applied to system model 141, particle states are generated by the strategy optimization module 144. The particle states generated by system model 141 are approximations of the state of the real system 11. Sensor model 142, derived from the particle states, calculates approximate particle measurements as measurements in the system module 10, and the model of the online state estimator 143, derived from the particle measurements and previous online particle estimates, calculates an online estimate. The online estimate is an approximation of the state estimate in the system module 10. The policy optimization block 144 takes online particle estimates and particle measurements as inputs and learns the optimal policy parameters for policy 14. During learning (training), policy optimization 144 generates control signals that are sent to the system model 141. When learning is complete, the policy parameters can be used to define policy 14 to control the real system.

[0054] Figure 1BA block diagram is shown illustrating a vehicle motion controller 100B configured to control the motion of a vehicle according to an embodiment of the present invention. In this configuration, the controller 100 can be applied to a vehicle control system to control the motion of the vehicle. The vehicle control system, including a model learning module and a policy learning module, is connected to the vehicle's motion controller and vehicle motion sensors that measure the vehicle's motion. The control system generates policy parameters based on the motion measurement data and provides these policy parameters to the vehicle's motion controller to update the policy unit of the motion controller.

[0055] The vehicle motion controller 100B may include an interface controller 110B, a processor 120, and a memory unit 130B. The processor 120 may be one or more processor units, and the memory unit 130B may be a memory device, a data storage device, etc. The interface controller 110B may be an interface circuit, which may include analog-to-digital (A / D) and digital-to-analog (D / A) converters for signal / data communication with the vehicle motion sensor 1101, road roughness sensor 1102, and motion controller 150B. Furthermore, the interface controller may include a memory to store data to be used by the A / D or D / A converters. The vehicle motion sensor 1101 and road roughness sensor 1102 are arranged on the vehicle to measure the vehicle's motion state. The vehicle includes a motion controller device / circuit that includes a strategy unit 151B to generate motion parameters to control the suspension system 1103, which controls suspension units 1103-1, 1103-2, 1103-3, and 1103-4. The suspension system can be 1103-1, 1103-2, 1103-3, ..., 1103-#N, depending on the number of wheels. For example, vehicle motion sensor 1101 may include an acceleration sensor, a positioning sensor, or a Global Positioning System (GPS) device to measure the vehicle's motion state. Road roughness sensor 1102 may include an acceleration sensor, a positioning sensor, etc.

[0056] The interface controller 110B is also connected to a vehicle motion sensor 1101 that measures the vehicle's motion state. Furthermore, the interface controller 110B can be connected to a road roughness sensor 1102 mounted on the vehicle to obtain information about the roughness of the road the vehicle is traveling on. In some cases, when the vehicle is an electric vehicle, the motion controller 150B can control the individual electric motors driving the vehicle's wheels. In some cases, the motion controller 150B can control the rotation of the individual wheels in response to policy parameters generated from the policy learning module 1400B to allow the vehicle to accelerate smoothly or decelerate safely. Additionally, depending on the design of the vehicle's driving operation, the motion controller 150B can control the wheel angles in response to policy parameters generated from the policy learning module 1400B.

[0057] Memory unit 130B may store computer-executable program modules including model learning module 1300B and policy learning module 1400B. Processor 120 is configured to execute the steps of program modules 1300B and 1400B. In this case, the steps may include offline modeling to generate an offline learning state using model learning module 1300B based on the vehicle's motion state (motion state) and measurement states from vehicle motion sensor 1101, road roughness sensor 1102, or a combination of vehicle motion sensor 1101 and road roughness sensor 1102. These steps also perform providing the offline state to policy learning module 1400B to generate policy parameters, and updating the policy 151B of the vehicle's motion controller 150B based on the policy parameters to operate the actuators or suspension system 1103.

[0058] Figure 1C A block diagram is shown illustrating a robot control system 100C for controlling the movement of a robot according to an embodiment of the present invention. The robot control system 100C is configured to control the robot's actuator system 1203.

[0059] In this configuration, controller 100 can be applied to robot control system 100C to control the robot's motion. Robot control system 100C, including model learning module 1300C and policy learning module 1400C, is connected to robot actuator controller 150C and sensor 1201 that measures the robot's motion. Robot control system 100C generates policy parameters based on motion measurement data, and provides these policy parameters to robot actuator controller 150C to update the actuator controller's policy unit 151C.

[0060] The robot control system 100C may include an interface controller 110C, a processor 120, and a memory unit 130C. The processor 120 may be one or more processor units, and the memory unit 130C may be a memory device, a data storage device, etc. The interface controller 110C may be an interface circuit that may include analog-to-digital (A / D) and digital-to-analog (D / A) converters for signal / data communication with the robot's sensors 1201 and motion controller 150C. Furthermore, the interface controller 110C may include memory to store data to be used by the A / D or D / A converters. The sensor 1201 is arranged at the joints of the robot (robot arm) or object-grabbing mechanism (e.g., a finger) to measure the robot's state. The robot includes an actuator controller (device / circuit) 150C, which includes a strategy unit 151C to generate motion parameters to control a robot system 1203. The robot system 1203 controls a robot arm, a manipulator, or a combination of arm and manipulator 1203-1, 1203-2, 1203-3, and 1203-#N based on the number of joints or manipulator fingers. For example, sensor 1201 may include an accelerometer, a positioning sensor, or a Global Positioning System (GPS) device to measure the vehicle's motion state. Sensor 1201 may include an accelerometer, a positioning sensor, etc.

[0061] The interface controller 110C is also connected to sensor 1201, which measures / acquires the motion state of the robot mounted on it. In some cases, when the actuators are electric motors, the actuator controller 150C can control the angle driving the robot arm or manipulate the individual electric motors of the object via a manipulator. In some cases, the actuator controller 150C can control the rotation of the individual motors arranged in the arm in response to policy parameters generated from the policy learning module 1400C, so as to smoothly accelerate or safely decelerate the robot's motion. Furthermore, depending on the design of the object manipulator, the actuator controller 150C can control the length of the actuator in response to policy parameters generated from the policy learning module 1400C.

[0062] Memory unit 130C may store computer-executable program modules including model learning module 1300C and policy learning module 1400C. Processor 120 is configured to execute the steps of program modules 1300C and 1400C. In this case, the steps may include offline modeling to generate an offline learning state using model learning module 1300C based on the robot's motion state and measurement states from sensor 1201. These steps also perform providing the offline state to policy learning module 1400C to generate policy parameters, and updating the policy 151C of the robot's motion controller 150C based on the policy parameters to operate actuator system 1203.

[0063] Figure 1D A schematic diagram is shown, illustrating a particle-based strategy optimization method for a control system according to an embodiment of the present invention.

[0064] Component 1301 represents a schematic diagram of the initial state distribution of the system. Components 1302 and 1312 are named particles and are embodiments of initial conditions sampled based on the initial state distribution. The particle state evolution of particle 1302 is represented by 1303, 1304, and 1305. Starting from 1302, the system model estimates the distribution of particle states in the following steps, which are represented by 1303 in the first step, 1304 in the second step, and 1305 in the third step. The state evolution continues for a duration determined by the number of simulation steps. Similarly, the particle state evolution of particle 1312 is represented by 1313, 1314, and 1315.

[0065] According to the present invention, some embodiments are described below. We state the general problems of model-based policy gradient methods and introduce a method for modeling dynamic systems using GP. Furthermore, we propose MC-PILCO, an algorithm for fully measurable systems, and describe in detail the policy optimization and model learning techniques employed. We analyze several aspects affecting the performance of MC-PILCO, such as cost shape, drop-off techniques, and kernel selection. In addition, we compare MC-PILCO with PILCO and Black-DROPS using a simulated cart-based inverted pendulum benchmark system, test the performance of MC-PILCO with a simulated UR5 robot, and also test the advantages of particle-based methods when dealing with different initial condition distributions. We propose extending the MCPILCO algorithm to systems with partially measurable states; this algorithm is now called MC-PILCO4PMS. Experiments will be shown as examples, with the controller according to the present invention applied to real Furuta pendulum and ball-and-disc systems.

[0066] Model-based policy gradient

[0067] In this section, we introduce the standard framework considered in model-based policy gradient RL methods, taking into account discrete-time systems described by unknown transformation functions f(·,·).

[0068] x t+1 =f(x) t ,u t )+w t ,

[0069] Where, at each time step t, and These are the system's state and input, respectively. It models additive noise using independent Gaussian random variables. The cost function c(x) t ) is defined as representing the state x. t Immediate punishment.

[0070] Based on the policy function that depends on the parameter vector θ Choose your input.

[0071] The goal is to find a strategy that minimizes the expected cumulative cost over a finite number of time steps T, i.e.

[0072]

[0073] The initial state is based on a given p(x0) distribution.

[0074] Model-based methods for learning strategies typically consist of a series of consecutive trials; that is, attempts to solve the desired task. Each trial consists of three main phases:

[0075] Model learning: Data collected from all previous interactions is used to build a model of the system's dynamics (in the first iteration, exploratory controls, which may be randomized, are applied to collect data);

[0076] Policy update: Optimize the policy based on the current model to minimize the cost J(θ);

[0077] Policy execution: Apply the currently optimized policy to the system and store the data for model improvement.

[0078] When the current policy is applied, model-based policy gradient methods use the learned model to predict state evolution. These predictions are used to estimate J(θ) and its gradient. In order to follow the gradient descent method to update the policy parameters θ.

[0079] • GPR and advance prediction

[0080] In this section, we discuss how to use Gaussian process regression (GPR) for model learning. We focus on three aspects: some background concepts about GPR, a description of model predictions one step ahead, and finally, we discuss long-term predictions, focusing on two possible strategies: moment matching and particle-based methods.

[0081] A common strategy in GPR-based methods is to model the evolution of each state dimension using different GPs. We use... This represents the difference between the values ​​of the i-th component at time t+1 and t, and is expressed as... express The noise measurement value, where i∈{1,…,d} x Furthermore, let This is a vector containing the state and input at time t, also known as the GP input. Then, given the data... in It is a vector of n output measurements, and It is the set of GP inputs. For each state dimension, GPR assumes the following probabilistic model:

[0082]

[0083] Where e (i) It has a standard deviation σ i The zero-mean Gaussian independent and identically distributed noise, and h (i) (·) is an unknown function that is a priori modeled as a zero-mean Gaussian process, and i∈{1,…,d} x In particular, we have Through kernel function k i (·,·) is the element-wise defined prior covariance matrix. That is, the elements in the j-th row and k-th column are determined by... Given. A key aspect of GPR is kernel selection. The kernel function encodes prior assumptions about the process. One of the most common choices for continuous functions is the SE kernel, defined as...

[0084]

[0085] The scaling factor λ and the matrix Λ are kernel hyperparameters that can be estimated by maximizing marginal likelihood. Typically, Λ is assumed to be diagonal, where the diagonal elements are named the length scale.

[0086] Significantly, h can be computed in closed-form. (i) The posterior distribution of (·). At time t, let This is a standard GP input.

[0087] Then, Distribution The estimate is a Gaussian with the mean and variance given below:

[0088]

[0089]

[0090] Among them, Γ i and Defined as

[0091]

[0092]

[0093] Recall modeling the evolution of each state dimension using different general GPs, and assuming that given the current GP input... In the case that GP is conditionally independent, then at time t+1, the estimated posterior distribution of the state at time t+1 is:

[0094]

[0095] in

[0096]

[0097]

[0098] Long-term forecasts using GP dynamic models

[0099] In MBRL, policy π is evaluated and improved based on long-term predictions of state evolution. θ : Considering the propagation of uncertainty, the precise calculation of these quantities requires the cascaded application of the GP model one step ahead. More precisely, starting from a given initial distribution p(x0), at each time step t, by... Marginalize (5) to obtain the next state distribution, i.e.

[0100]

[0101] Unfortunately, calculating the exact predicted distribution in (8) is not easy. There are different methods to approximate this problem, and here we discuss two main approaches: moment matching adopted by PILCO and particle-based methods, which are the strategies followed in this work.

[0102] Moment matching

[0103] Assuming the GP model uses only the SE kernel as the prior covariance and considers the normal initial state distribution x0:N(μ0,Σ0), it can be computed in closed-form. The first and second moments. Then, the distribution. The distribution is approximated as Gaussian, with its mean and variance corresponding to the previously calculated moments. Finally, the subsequent probability distribution is calculated iteratively for each time step in the prediction time domain. Details regarding the calculation of the first and second moments are provided. The advantage of moment matching is that it provides a closed-form solution through the GP dynamic model, handling uncertainty propagation. Therefore, in this setting, the policy gradient can be calculated analytically from long-term predictions. However, as mentioned above, the Gaussian approximation performed in moment matching is also the cause of two major drawbacks: (i) the assumption of using the SE kernel and the calculation of both moments can lead to poor generalization properties in data not seen during training; and (ii) moment matching allows modeling only a unimodal distribution, which can be an overly restrictive approximation of the real system behavior.

[0104] Particle-based methods

[0105] The integral in (8) can be approximated using Monte Carlo methods, particularly particle-based methods. Specifically, M particles are sampled from the initial state distribution p(x0). Each of the M particles is propagated using the one-step-ahead GP model (5). The state of the m-th particle at time t is set as... Where m = 1, ..., M. At time step t, the actual policy π is evaluated. θ To calculate the associated control. The GP model provides a Gaussian distribution. From this, the state of the particle in the next time step. Sampling is performed. This process is iterated until a trajectory of length T is generated for each particle. For clarity, this process is... Figure 1D The diagram shows that the long-term distribution at each time step approximates the particle distribution. Note that this method imposes no constraints on the choice of the kernel function or the initial state distribution. Furthermore, it does not specify the distribution... An approximation is made. Therefore, the particle-based approach is unaffected by the problems seen in moment matching, at the cost of greater computational complexity. Specifically, the computation of (5) requires the computation of (3) and (4), where (3) and (4) are the mean and variance of the incremental states, respectively. Regarding computational complexity, it can be noted that Γ is calculated offline in a single step during the training of the GP model. i -1 y (i) (The same computation is required in the moment matching case), and the number of operations required to compute (3) is linear with respect to the number of samples n. The computational bottleneck is the computation of (4), which is O(n^2). 2 Then, the cost of predicting a single state is O(d). xn 2 ), leading to O(d x MTn 2 The total computational cost depends on the complexity of the system dynamics; the number of particles required to obtain a good approximation can be high, thus determining a considerable computational burden. However, the computational burden can be largely mitigated by parallel computing via GPUs, as the evolution of each particle can be computed in parallel.

[0106] MC-PILCO

[0107] In the following section, we present an algorithm for fully measurable systems. MC-PILCO relies on the GPR used for model learning and follows a Monte Carlo sampling method to estimate the expected cumulative cost based on the particle trajectories propagating through the learned model. We utilize a reparameterization technique to obtain the policy gradient from the sampled particles and optimize the policy. This approach is highly flexible and allows the use of any kind of kernel for the GP, providing a more reliable approximation of the system behavior. MC-PILCO generally consists of iterative steps of three main steps: updating the GP model, updating the policy parameters, and implementing the policy on the system. In this order, the policy update consists of three steps, for N... opt Iterate over the maximum value:

[0108] Based on the current π θ And GP models learned from previously observed data to simulate particle evolution;

[0109] An approximation of the expected cumulative cost is calculated based on the evolution of M particles.

[0110] based on Update the policy parameter θ. It's about θ The gradient is calculated through backpropagation.

[0111] In the following sections, we will discuss the model learning steps and policy optimization steps in more detail.

[0112] Model learning program

[0113] Here, we describe the model learning framework considered in MC-PILCO. We first present the proposed one-step prediction model. Then, we discuss the choice of kernel function. Finally, we briefly discuss the hyperparameter optimization of the model and strategies for reducing computational cost.

[0114] One step ahead model

[0115] Let the state be defined as in It is a vector of the system's generalized coordinates at time step t, and express The derivative with respect to time. MC-PILCO employs a one-step advance model, which is subsequently expressed as a velocity integral dynamic model, utilizing the state components q and The inherent correlation between them. In fact, when considering a sufficiently small sampling time T... s When (small for the application), assuming a constant acceleration between two consecutive time steps is reasonable, q is obtained. t The following evolution:

[0116]

[0117] Let I q (respectively ) and q (respectively) The proposed velocity integral model learns only d, which is an ordered set of dimensional indices of the associated states x. x / 2GP, Evolution of each GP for different velocity components Modeling is performed, among which Then, the evolution of the position change is calculated based on (9) and the predicted velocity change. Among them, i k ∈I q Many previous MBRL algorithms adopted the standard model described above, and this has since been expressed as a full-state dynamic model. The full-state model uses distinct and independent general GPs to predict the changes in each state component. This is done by assuming that the evolution of each state dimension is conditionally independent given the current GP input and needs to be learned to be equal to the state dimension d. x The number of GPs is reduced. Then, compared to the full-state model, the proposed velocity integral model halves the number of GPs to be learned, thereby reducing the cost of state prediction to [amount missing]. However, this method is based on the assumption of constant acceleration and only works appropriately when considering sufficiently small sampling times. MC-PILCO, however, can also use a standard full-state model, which may be more effective when the sampling time is too high.

[0118] Kernel function

[0119] Regardless of the GP dynamic model structure used, one advantage of particle-based policy optimization methods is the ability to choose any kernel function without restriction. Therefore, we consider different kernel functions as examples of modeling the evolution of physical systems. However, readers may consider custom kernel functions suitable for their applications.

[0120] • Squared exponent (SE). The SE kernel described in (2) represents the standard choice used in many different works.

[0121] SE+ polynomial (SE+P) (d)Recalling that the sum of kernels is still a kernel, we also consider kernels given by the sum of SE and polynomial kernels. Specifically, we use multiplicative polynomial (MP) kernels, which are refinements of the standard polynomial kernel. An MP kernel of degree d is defined as the product of d-linear kernels, i.e.:

[0122]

[0123] in The matrices are distinct diagonal matrices. diagonal elements and The elements together constitute the kernel hyperparameter.

[0124] The result is as follows

[0125]

[0126] The idea behind incentivizing this choice is as follows: the MP core allows capture as The possible patterns of systems with polynomial functions in the polynomial kernel are typical of mechanical systems, while the SE kernel models more complex behaviors that are not captured by the polynomial kernel.

[0127] • Semiparametric (SP). When prior knowledge about the system dynamics is available, such as from first principles of physics, a so-called physical excitation (PI) kernel can be derived. The PI kernel is derived from appropriate basis functions. The linear kernel defined above. More precisely, GP input determined by the physical model The possible nonlinear transformation. Therefore, we have:

[0128]

[0129] Where ∑ PI yes A positive definite matrix whose elements are k PI Hyperparameters; to limit the number of hyperparameters, standard choices include ∑ PI It is considered as a diagonal. To compensate for potential inaccuracies in the physical model, k is usually... PI Combined with the SE kernel, a so-called semi-parametric kernel is obtained, expressed as:

[0130]

[0131] The fundamental principle behind this core is as follows: k PI The encoding is based on prior information given by physics, and k SE Compensation k PI Unmodeled dynamic components.

[0132] Model optimization and reduction techniques

[0133] In MC-PILCO, the generalization process (GP) hyperparameters are optimized by maximizing the marginal likelihood (ML) of the training samples. Previously, we recognized that the computational cost of particle prediction is proportional to the square of the number of samples, n, leading to a high computational burden as n increases. Therefore, strategies to limit the computational burden of prediction must be implemented. Several solutions have been proposed in the literature. We implement a procedure in which the authors propose an online importance sampling strategy. After optimizing the GP hyperparameters by maximizing ML, samples in D are downsampled to a subset. Then it is used to calculate predictions. The process first uses the first sample in D to calculate predictions. r It performs initialization, and then uses D. r The GP estimate is iteratively computed for all remaining samples in D, using them as training samples. If the uncertainty of the estimate exceeds a threshold β... (i) Then add the samples in D to D. r Or discard it. Each time a sample is added to D r Update the GP estimator in real time. By tuning β (i) This is to balance the trade-off between reducing computational complexity and the severity of the introduced approximations. β (i) The higher, D r The smaller the sample size, the better. On the other hand, using β... (i) Too high a value may impair the accuracy of GP predictions.

[0134] Strategy Optimization Program

[0135] Here, we present the policy optimization strategy employed in MC-PILCO. We begin by describing the general policy structure under consideration. Later, we show how to estimate the policy gradient from long-term particle-based predictions using backpropagation and reparameterization techniques. Finally, we explain how dropout is implemented within this framework.

[0136] Strategy Structure

[0137] In all the experiments presented in this work, we consider the RBF-network strategy, whose output is constrained by a hyperbolic tangent function with appropriate scaling. We call this function squared-RBF-network and define it as follows:

[0138]

[0139] The policy parameters are θ = {w, A, ∑ π},in and These are the weights and center of the Gaussian function, respectively, and ∑ π Determine the shape of the Gaussian function; in all experiments, we assume ∑ πIt's diagonal. Maximum control action u max It is constant and selected based on the system to be controlled. It is worth noting that MC-PILCO can handle any differentiable policy, and therefore more complex functions such as deep neural networks can also be considered.

[0140] gradient calculation

[0141] MC-PILCO derives the policy gradient by applying a reparameterization technique to the computation of the estimated expected cumulative cost obtained from (1) which depends on Monte Carlo sampling. Given a control policy π θ Given the initial state distribution p(x0), as described above, the evolution of a sufficiently high number of particles is predicted. Therefore, the sample mean of the cost caused by a particle at time step t approximates each Specifically, let Let be the state of the m-th particle at time t, where m = 1, ..., M and t = 0, ..., T. The Monte Carlo estimate of the expected cumulative cost is calculated using the following expression.

[0142]

[0143] To compute the gradient of (12) with respect to the policy parameters, we use a reparameterization trick to distinguish it by random operations. Defined from (6)-(7) The evolution of each particle at the next time step is sampled using a normal distribution. However, instead of directly from N(μ) t+1 ,Σ t+1 The sampling technique involves reparameterizing a point ε from a zero-mean, unit-variance normal distribution of appropriate dimensions. This point is then mapped to the desired distribution. Among them, L t+1 It is ∑ t+1 The Cholesky decomposition, i.e. Now, we can distinguish based on the distribution parameters. and We simply compute this via backpropagation. We update the policy parameters using the Adam solver; we will use α... lr This indicates Adam's step size.

[0144] Disposal technology

[0145] To improve the exploration of parameter θ and increase the ability to escape local minima during policy optimization, we consider using a drop-off technique. Assuming the policy is the squashed-RBF-network in (11), the process employed is described; similar considerations can be applied to different policy functions. When the drop-off technique is applied to the policy in (11) at each evaluation of the policy function, with probability p...d The random drop policy weights w are determined. This operation is performed using a random variable r. i ~Bernoulli(1-p d Scaling each weight w i To execute, where Bernoulli(1-p d ) represents the Bernoulli distribution, with the assumed value 1 / (1-p) d The probability of () is 1-p d And the probability of 0 is p. d This operation is equivalent to defining the probability distribution of w, thereby obtaining a parameterized stochastic policy. Specifically, for each w... i The distribution of can be approximated by a bimodal distribution, which is characterized by a bimodal distribution with infinitesimal variance ξ. 2 The sum of two appropriately scaled Gaussian distributions is defined as follows:

[0146]

[0147] Using a stochastic policy during policy optimization allows for an increase in the entropy of the particle distribution. This property increases the probability of accessing low-cost regions and escaping local minima. Furthermore, we validate that the dropout technique can mitigate problems associated with exploding gradients. This is likely due to the fact that the gradient is computed using the average of several distinct values ​​of w, rather than a single value of w; i.e., using different policy functions, thus achieving regularization of the gradient estimate.

[0148] In contrast, the use of a stochastic policy can affect the accuracy of the obtained solution due to the added entropy. We also need to consider that the ultimate goal is to obtain a deterministic policy. For these reasons, we design a heuristic scaling process to gradually reduce the dropout rate p. d Until it equals 0. The scaling action is determined by... The monitoring signal s is triggered by the statistical definition of past history. Cost changes are defined. Where θ j Let represent the policy parameters at the j-th optimization step. Then, s is calculated as... and A filtered version of the ratio between them. and These are calculated using an exponential moving average (EMA) filter. The mean and standard deviation of . The expression for s in the j-th optimization step is as follows:

[0149]

[0150]

[0151]

[0152] Where α s These are the coefficients of the exponential moving average filter, determining the filter's memory. During each iteration of the optimization process, the algorithm checks the last n... s Is the absolute value of the monitoring signal s during iteration lower than the threshold σ? s ,Right now

[0153]

[0154] Where < is the element-wise operator, and the condition in (14) is true if all elements are verified. If the condition is verified, then by quantity Δp d Decrease p d And the optimizer's learning rate α lr and σ s Both are based on arbitrary factors λ s Scaling. Therefore, we have:

[0155] p d =p d -Δp d (15a)

[0156] α lr =λ s α lr (15b)

[0157] σ s =λ s σ s (15c)

[0158] if only Then the process is iterated.

[0159] in It is the minimum value of the learning rate.

[0160] The basic principle behind this heuristic scaling process is as follows. If Close to zero, or if If it is particularly high, then s j The signal is small. The first case occurs when the optimization reaches its minimum, and high variance indicates that the particle's trajectory intersects with regions of the workspace where the uncertainty of the GP prediction is high. In both cases, we test the strategy on a real system: the first case to verify whether the configuration solves the task, and the second case to collect data on prediction uncertainty, thereby improving model accuracy. The algorithm MC-PILCO with dropout techniques is summarized in... Figure 1F In the pseudocode.

[0161] Unless otherwise expressly stated, we use Figure 1EThis report concludes the discussion on policy optimization by showing all the optimized parameters used in the proposed experiments. However, it is worth noting that some adaptation may be required in other settings, depending on the problem being considered.

[0162] Ablation Research

[0163] In the following sections, we analyze several aspects affecting the performance of MC-PILCO, such as the shape of the cost function, the use of the dropout technique, kernel selection, and the probabilistic model adopted (i.e., the full-state or velocity integral dynamic model). The purpose of the analysis is to verify the choices made in the proposed algorithm MC-PILCO and to demonstrate their effectiveness in dynamic system control. MC-PILCO has been implemented in Python, utilizing the automatic differentiation function of the PyTorch library; the code is publicly available. We consider simulating the swing of a cart-inverted pendulum (a classic benchmark problem) to perform an ablation study. The system and experiments are described below. The physical properties of the system are the same as those used in PILCO: the mass of the cart and pendulum is 0.5 [kg], the length of the pendulum is L = 0.5 [m], and the coefficient of friction between the cart and the ground is 0.1. The state at each time step t is defined as... Where p t Let θ represent the position of the cart, and θ t This represents the angle of the pendulum. The target state corresponding to the swing of the pendulum is determined by p. des =0[m] and |θ des |=π[rad] is given. At θ t =0 [rad] defines a downwardly stable equilibrium point. To avoid singularities caused by angles, states are used in the algorithm. Replace x t The controlling action is the force that pushes the cart horizontally. In all the following experiments, we will have 10 -2 The white measurement noise of the standard deviation is regarded as the initial state distribution N([0,0,0,0],diag([10)). -4 10 -4 10 -4 10 -4 The sampling time is 0.05 seconds. The strategy is to have n b A squashed-RBF network with 200 basis functions. It receives... and u max =10[N] as input. The number of particles has been set to M=400 in all tests. The exploration trajectory is obtained by applying random control actions sampled from U(-10,10) at each time step t. GP reduction technique has not yet been adopted.

[0164] The cost function optimized in MC-PILCO is as follows:

[0165]

[0166] Among them l θ and l p This is named the length scale. Note that the length scale defines the shape of c(·), and the cost function reaches its maximum value more quickly with a smaller length scale. Therefore, higher costs are associated with lower l. θ and l p The target states are associated with the same distance. The lower the length scale, the greater the selectivity of the cost function. θ is required. t The absolute value of the value is used to allow for different pendulum start solutions for the equivalent target angles π and -π of the pendulum.

[0167] All comparisons included a Monte Carlo study consisting of 50 experiments. Each experiment comprised 5 trials, each lasting 3 seconds. The random seed changed in each experiment, corresponding to different policy explorations and initializations, as well as different implementations of measurement noise. The performance of the learning policy was evaluated using cost.

[0168]

[0169] in It is the squared distance between the tip of the pendulum and its position at the unstable equilibrium point, where p t =0[m]. We introduce this cost to provide a common metric for comparing different settings of MC-PILCO and other MBRL algorithms. For each trial, we report the cost per unit time (C) using c. pilco (·) The median and confidence intervals defined by the fifth and ninety-fifth percentiles of the obtained cumulative cost, and the observed success rate. We label two values ​​of cumulative cost associated with the optimal and suboptimal pendulum swings, respectively. The solution is optimal if the pendulum oscillates only once before reaching upward equilibrium. The solution is suboptimal when the pendulum oscillates twice. Finally, if |p| in the last second of the trial t |<0.1[m]and 170[deg]<|θ t If |<190[deg], then we will mark the experiment as "successful".

[0170] Cost shaping

[0171] The first test concerns the performance obtained by changing the length scale of the cost function in (19). Backward shaping is a known important aspect of RL, and here we will analyze it in relation to MCPICO. Figure 2A and Figure 2B In the middle, we make comparisons using (l) θ =3,l p =1) and (l θ =0.75,lp The evolution of the cumulative cost obtained (=0.25) is presented, and we report the observed success rate. The latter length scale defines a more selective cost as the function shape becomes more skewed. In both cases, we employ a velocity integral model with an SE kernel and do not use dropout techniques during policy optimization.

[0172] Figure 2A and Figure 2B The results show that, using (l) θ =3,l p In the case of (l = 1), MC-PILCO performs better. In fact, when utilizing (l θ =0.75,l p With (=0.25), MC-PILCO found solutions only in 74% of the experiments, while using (l θ =3,l p With (l) the success rate is 82%. Observing the cumulative cost also helps to understand the difference in the quality of the learned strategy in the two cases. Only when using (l) θ =3,l p Only when (=1) can the optimal pendulum setup be found in most experiments, while using (l) θ =0.75,l p =0.25) The optimal swing was not achieved.

[0173] This fact suggests that using an overly selective cost function can significantly reduce the probability of convergence to a solution. The reason may be that when the policy parameters are far from a good configuration, c(x) becomes less efficient at smaller length scales. t The length scale is very sharp, resulting in a near-zero gradient and increasing the probability of getting trapped in a local minimum. Conversely, a larger length scale promotes the existence of non-zero gradients outside the objective, which is beneficial to the policy optimization process. These observations have been made in PILCO, but they did not encounter difficulties when using small length scales such as 0.25 in (20). This may be due to moment matching and the different optimization algorithms used, which make it possible to perform policy gradient analysis. On the other hand, the value of the length scale does not seem to affect the accuracy of the learned solution. To confirm this, in Figure 6C Lines 3 and 4 report the average distance to the target state obtained at point 5 using the successful strategy during the last second of the interaction. No significant difference in accuracy was observed upon reaching the target.

[0174] Disposal technology

[0175] In this test, we compare the results obtained with and without the dropout technique during policy optimization. Figure 3A and Figure 3BIn this paper, we compare the evolution of the cumulative costs obtained in two cases, and we show the success rate obtained.

[0176] In both scenarios, we employ a velocity integral model with an SE kernel and a model with a length scale (l θ =3,l p =1) cost function. When using the dropout technique, MC-PILCO learns the optimal solution for trial 4 in 94% of the trials and manages to obtain the optimal solution for all random seeds through trial 5. Conversely, without the dropout technique, the optimal strategy is not always found even in the last trial. Note that without the dropout technique, the upper limit of the cumulative cost in the last two trials is higher, and the task cannot always be solved. Additionally, Figure 6C Lines 2 through 4 show that using the discarding technique also helps reduce the trolley positioning error at the end of the swing (in terms of mean and standard deviation).

[0177] Based on our experience, we have found that the dropout technique not only helps stabilize the learning process and find better solutions more consistently, but also improves the accuracy of the learned strategy.

[0178] Kernel function

[0179] In this test, we compared using SE or SE+P. (2) The results obtained from the kernel. In both cases, we employ a velocity integral model, using a length scale (l) θ =3,l p =1) Define the cost function and use the dropout technique. Figure 4A and Figure 4B Show SE+P (2) It converges to the optimal solution faster than SE. Using SE+P (2) The first kernel, the algorithm, learns the task in trial 3 90% of the time and achieves 100% success in trial 4. On the other hand, when using the SE kernel, the task is solved only in trial 5 for all random seeds. This can be explained by the ability of a more structured kernel to learn sufficiently accurate dynamics of the system, even in regions of the state-action space where no data points are available. In fact, some parts of the dynamics of the cart-inverted pendulum system are GP inputs. A polynomial function, and SE+P (2) The structure improves the data efficiency of model learning.

[0180] Velocity integral model

[0181] In this test, we compare the performance obtained by the proposed velocity integral dynamic model and the standard global state model. In both cases, the SE kernel is chosen, and the length scale (l) is used. θ =3,lp =1) Define the cost function and use the dropout technique. Figure 5A and Figure 5B The results show that the velocity integral model achieves better performance in experiments 2 and 3, with narrower confidence intervals and a better success rate. Conversely, the full-state model has a slightly better success rate in the last two experiments. Recall that in the full-state model, position and velocity are learned independently, while in the velocity integral model, position is calculated as an integral of velocity under the assumption of constant acceleration. Therefore, the velocity integral model can reduce uncertainty in long-term predictions and facilitates learning about corresponding pairs when few data points are collected. In fact, the full-state model may face some difficulties in learning the connection between position and corresponding velocity from a limited amount of data. This reduction in uncertainty can explain the narrower confidence intervals observed in the first few experiments. On the other hand, when enough data points have been collected (experiments 4 and 5), the improvement in accuracy obtained by the full-state model is not very significant. Even with comparable performance, the choice of the velocity integral model is reasonable because it halves the number of GPs to be learned, thus improving computation time.

[0182] Experiment in simulation

[0183] In the following sections, two simulation systems are considered. First, MC-PILCO is tested on a cart-based inverted pendulum system and compared with other policy gradient algorithms (i.e., PILCO and Black-DROPS). In the same environment, we test MC-PILCO's ability to handle bimodal probability distributions. Second, MC-PILCO learns a controller in the joint space of an UR5 robot arm, which is considered an embodiment of a higher DoF system.

[0184] Inverted pendulum with a trolley: A comparison with other methods

[0185] We tested PILCO, Black-DROPS, and MC-PILCO on the previously described inverted pendulum system for trolleys. In MC-PILCO, we considered systems with a length scale (l θ =3,l p =1) and the cost function of the SE kernel (19) to have the same kernel function in all three algorithms. The cumulative cost results are reported in Figure 6A and Figure 6BIn the middle, MC-PILCO achieved best performance in both temporality and convergence, learning in Experiment 5 how to swing the cart upside down with 100% success rate. In each trial, MC-PILCO achieved a cumulative cost with a lower median and less variability. On the other hand, the strategy in PILCO showed poor convergence characteristics, with only a 42% success rate after all 5 trials. Black-DROPS outperformed PILCO, but it achieved worse results than MC-PILCO in each trial, with only an 86% success rate in Experiment 5. Recall that MC-PILCO, considering SE+P... (2) It can achieve even better performance when the core is in use. Figure 6C The results (rows 1, 2, 6, and 7) also show that the strategy learned using MC-PILCO is more accurate in reaching the target.

[0186] Inverted pendulum cart: handling bimodal distribution

[0187] One of the main advantages of particle-based policy optimization is its ability to handle multimodal evolutions. This is impossible when applying moment-matching based methods such as PILCO, especially considering the very high variance of the initial cart position. We validate this advantage by applying both PILCO and MC-PILCO to a simulated inverted pendulum cart system, where the very high variance of the initial cart position corresponds to an unknown initial position of the cart (but within reasonable limits). The aim is that, in such cases, the strategy must solve the task regardless of the initial conditions and needs to exhibit bimodal behavior to achieve optimal results. Note that the described situation may be meaningful in several practical applications. We maintain the same setup used in previous inverted pendulum cart experiments, changing the initial state distribution to a zero-mean Gaussian diag([0.5,10) with a covariance matrix. -4 10 -4 10 -4 MC-PILCO utilizes length scales (l) θ =3,l p =1) Optimize the cost in (19). We test the strategies learned by the two algorithms starting from nine different initial cart positions (-2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2 [m]). Previously, we observed that PILCO had difficulty converging to a solution, and the high variance of the initial conditions highlighted this problem. However, in order to be able to compare, we preferentially selected a random seed for which PILCO converged to a solution in this specific scenario. Figure 7A and Figure 7BThe experimental results are shown in the figure. MC-PILCO is able to handle high initial variance. It learns a bimodal strategy: pushing the cart in two opposite directions depending on its initial position, and this stabilizes the system in all experiments. In contrast, PILCO's strategy cannot control the initial conditions of the inverted pendulum cart for all tests. Its strategy always pushes the cart in the same direction, and it fails to stabilize the system when the cart starts to move away from the zero position. The state evolution under MC-PILCO's strategy is bimodal, while PILCO cannot find this type of solution due to the unimodal approximation enforced by moment matching.

[0188] In this embodiment, we have seen that, due to its dependence on initial conditions, multimodal evolution can be optimal when starting from a unimodal distribution with high variance. In other cases, multimodal evolution can be directly enforced by the existence of multiple possible initial conditions that are difficult to model using a single unimodal distribution. MC-PILCO can handle all of these cases thanks to its particle-based long-term prediction method. Similar results were obtained when considering bimodal initial distributions. Due to space limitations, we do not report the results obtained, but the experiments can be found in the code in the supplementary material.

[0189] UR5 Joint Space Controller: High DoF Applications

[0190] The goal of this experiment is to test MC-PILCO in a more complex system with higher DoF. We use MC-PILCO to learn the joint space controller of the UR5 robotic arm (6DoF) simulated in MuJoCo. Let the state at time t be... Where q t , These are joint angle and velocity, respectively. Strategy π θ The goal is to control the torque τ t To follow the expected trajectory make Let n be the position and velocity errors at time t, respectively. The strategy is defined in (11) with n... b =400 Gaussian function and u for all joints max =1 [N·m] multi-output squashed-RBF-network, which maps state and error to torque. Control scheme is expressed in Figure 8 middle.

[0191] In this experiment, we consider a 4-second control time domain with a sampling time of 0.02 seconds. A reference trajectory has been calculated to allow the end effector to draw a circle in the XY operating space. Initial exploration for initializing the velocity integral dynamic model is provided by a minimally tuned PD controller. We use SE+P in the GP dynamic model. (1) The core. The GP reduction threshold is set to 10. -3 Using extended state Construct the GP input. M = 200 is the number of particles used for gradient estimation. The cost function under consideration is defined as:

[0192]

[0193] We assume full-state observability, where the measurements are defined by 10... -3 White noise perturbation with standard deviation. The initial state distribution is based on 10... -3 Standard deviation A Gaussian distribution centered at n. s =400 and δ s Beyond 0.05, the strategy optimization parameters and Figure 1E The same applies to the report, but with more restrictive exit conditions.

[0194] exist Figure 9A and Figure 9B In this paper, we report the trajectory followed by the end effector in each test and the expected trajectory. MC-PILCO significantly improves upon the high tracking error obtained with the PD controller after only 2 tests (corresponding to 8 seconds of interaction with the system). The learned control strategy follows the reference trajectory of the end effector with a mean error of 0.65 ± 0.69 [mm] (confidence intervals calculated as 3 × standard deviation) and a maximum error of 1.08 [mm].

[0195] MC-PILCO for partially measurable systems

[0196] In the following sections, we discuss applying MC-PILCO to partially measurable systems, i.e., systems whose states are observable, but only some components of the state can be directly measured, while the remaining components must be estimated from the measurements. For simplicity, we introduce the case of mechanical systems where only position (not velocity) can be measured, but a similar consideration can be made for any partially measurable system with observable states. We then describe MC-PILCO for partially measurable systems (MC-PILCO4PMS), a variant of MC-PILCO proposed to handle such settings. As a proof of concept, the algorithm MC-PILCO4PMS is validated in simulations.

[0197] MC-PILCO4PMS

[0198] Consider a mechanical system where only joint position can be measured. This can be described as a partially measurable system, where in state... Below, only q is measured. t Therefore, through an appropriate estimation process, from q t Measurement history estimation The element may still perform q under conditions of high measurement noise. t The denoising operation is crucial. It's particularly important to distinguish between online and offline estimations. The former are provided to the control policy to determine the system control inputs, and they must adhere to real-time constraints—that is, the velocity estimates must be causal and computation must be performed within a given interval. The latter do not need to handle such constraints. Therefore, considering non-causal information and limiting delays and distortion, offline estimations can be more accurate.

[0199] In this context, we validate the importance of distinguishing between particle state predictions computed by the model and the data provided to the policy during policy optimization. In practice, GPs should simulate real system dynamics independently of additional noise from sensing instruments, thus requiring them to utilize the most accurate available estimates to function; latency and distortion can impair long-term prediction accuracy. On the other hand, directly providing the policy with particle states computed using GPs during policy optimization is equivalent to training the policy under the assumption of direct access to the system state, which, as previously mentioned, is impossible in the considered setting. In fact, the considerable difference between the particle states and the state estimates computed online during policy application to the real system can compromise the policy's effectiveness. This approach differs from standard MBRL methods, which typically do not consider the influence of online state estimators during training.

[0200] To address the aforementioned issues, we introduced MC-PILCO4PMS, a variant of MC-PICO. Within MC-PILCO4PMS, we introduce the following additional features related to MC-PICO.

[0201] Calculate GP training data using an offline state estimator

[0202] We compute the state estimates used to train the GP model using an offline state estimator. Specifically, in our real-world experiments, we consider two computational options with a central difference formula, i.e. Where T s This refers to the sampling time. This technique can only be used when measurement noise is limited; otherwise... It's probably too noisy.

[0203] The state is estimated using a Kalman smoother, where the state-space model is given by general equations relating to position, velocity, and acceleration. The advantage of this technique is that it utilizes the correlation between position and velocity, thus increasing regularization.

[0204] Simulation of online estimators

[0205] During policy optimization, we simulate not only the evolution of particle states, but also a measurement system with a sensor model and an online estimator with an online estimator model. The state fed to the measurement (denoted as z) t The state is calculated similarly to the state estimated online. Given the m-th particle, this is:

[0206]

[0207] Where f z This represents an online state estimator, which has a memory m. q and m z ,and This is a virtual noise measurement of the position of the m-th particle. More precisely, let... for Given the position of the particle's state, we have:

[0208]

[0209] in It has a mean of zero and a covariance of The noise is Gaussian independent and identically distributed. It must be adjusted according to the characteristics of the measurement system (e.g., encoder accuracy). The value. Then, the control input for the m-th particle is calculated as... Instead The differences in particle generation between MC-PILCO and MC-PILCO4PMS are summarized in Figure 10A and Figure 10B The flowchart in the report.

[0210] Simulation results

[0211] Here, we test the relevance of modeling the existence of an online estimator using a simulated cart-and-pendulum system, but with the added assumption of simulating a real-world experiment. We consider the same physical parameters and initial conditions described for the previously described cart-and-pendulum system, but assume that only the cart position and pendulum angle are measured. We model the measurement system that might exist in real life as having a standard deviation of 3.10. -3The additive Gaussian independent and identically distributed noise was used. Samples were collected at 30 Hz to obtain a reliable estimate of the velocity. The online velocity estimate was calculated using causal numerical differentiation followed by a first-order low-pass filter with a cutoff frequency of 7.5 Hz. The velocity used to train the GP was derived using the central difference formula. The effectiveness of MC-PILCO4PMS associated with MC-PILCO was validated in this system. Exploration data was collected using a random exploration strategy. To avoid dependence on initial conditions (e.g., policy initialization and exploration data), the same random seed was fixed in both experiments. Figure 11A and Figure 11B In this paper, we report the results of 400 runs of the Monte Carlo study. Figure 11A Above, the final policy is applied to the learned model (ROLLOUT), and... Figure 11B The final policy was applied to the inverted pendulum system (TEST). Even though the two policies performed similarly when applied to the model (which can be tested offline), the results obtained by testing the policy in the push-pole system were significantly different. MC-PILCO4PMS solved the task in all 400 attempts. Conversely, in several attempts, MC-PILCO failed to solve the task because the online filter introduced latency and discrepancies, which were not considered during policy optimization. We believe that these considerations regarding how to manipulate data during model learning and policy optimization would be beneficial for other MBRL algorithms different from MC-PILCO.

[0212] Experiments using the example system

[0213] In the following sections, we test MC-PILCO4PMS when it is applied to a real system. Specifically, we conduct experiments on two benchmark systems: Figure 12A Furuta pendulum and Figure 12B The ball-and-socket joint. These are merely a few examples of real-world systems to which some embodiments of the invention can be applied. Other examples of real-world systems could be robot manipulators, vehicles, and suspension systems.

[0214] Furuta pendulum

[0215] The Furuta pendulum (FP) is a popular benchmark system used for nonlinear control and reinforcement learning. The system consists of two rotary joints and three links. The first link (called the base link) is fixed and perpendicular to the ground. The second link (called the arm) rotates parallel to the ground, while the axis of rotation of the last link (the pendulum) is parallel to the principal axis of the second link. See [link to relevant documentation]. Figure 12AThe FP is an underactuated system because only the first joint is actuated. Specifically, in the FP considered, the horizontal joint is actuated by a DC servo motor, and both angles are measured by an optical encoder with 4096 [ppr]. The control variable is the motor voltage. Let the state at time step t be... in It is the angle of the horizontal joint, and It is the angle of the vertical joint attached to the pendulum. The goal is to learn a controller that enables the pendulum to swing and stabilize in upward equilibrium. Among them [rad]. The test length is 3 seconds, and the sampling frequency is 30 [Hz]. The cost function is defined as:

[0216]

[0217] in

[0218]

[0219] The first part of the function in (20) aims to move towards and Drive two angles simultaneously, c b (x t The penalty solution, in which or If the horizontal joints rotate too much, we set these boundaries to avoid the risk of damaging the system. The offline state estimator uses central difference to compute an offline estimate of the velocity of the GP model. For the online state estimator, we use causal numerical differentiation: Where T s This is the sampling time. We consider the extended state in the GP input. Instead of considering x t The strategy unit is equipped with online state estimation for received particles. n as input b A quadshed-RBF network with 200 basis functions. We use 400 particles to estimate the policy gradient based on model predictions. The exploration trajectory is obtained using the sum of ten sine waves with random frequencies and the same amplitude as input. The initial state distribution is assumed to be N([0,0,0,0]). T ,diag([5·10 -3 ,5·10 -3 ,5·10 -3 ,5·10 -3 M=400 particles were used for gradient estimation, and the GP reduction threshold was set to 10. -3We use three different choices of the kernel function mentioned above to solve the task: squared exponent (SE), squared exponent + d degree polynomial (SE + P). (d) ) and semi-parameter (SP). In Figure 13A and Figure 13B The diagram shows the trajectory obtained from each trial.

[0220] The MC-PILCO4PMS has managed to learn how to swing the Furuta pendulum in all conditions. It performs this in experiment 6 with a core SE, and in experiment 6 with a core SE+P. (2) Experiment 4 and Experiment 3 with SP kernels were successful. These experimental results confirm the higher data efficiency of more structured kernels and the advantages provided by MC-PILCO4PMS by allowing any kind of kernel function. Furthermore, we can observe that the cost function (20) always remains constant in all experiments and for any kernel tested. Maintaining effectiveness within the desired boundary. Considering c, which is similar to the value inside the cost function. b (x t The penalty is sufficient to handle soft constraints in other scenarios. Videos of the FP experiments are available in the supplementary materials.

[0221] Ball plate

[0222] The ball-disc system consists of a square plate that can be tilted in two orthogonal directions using two motors. A camera on top tracks the ball and measures its position on the plate. Let be the position of the center of the sphere along the X and Y axes, and and It is the angle by which the two motors tilt the plate at time t. Therefore, the state of the system is defined as... The motor driver only allows position control and does not provide feedback on the motor angle. To track the motor angle, we define the control action as the difference between two consecutive reference values ​​sent to the motor controller, and we limit the maximum input to a sufficiently small value so that the motor controller can reach the target angle within the sampling time. Then, in the first approximation, the reference angle and the motor angle coincide, and we have and The goal of the experiment is to learn how to control the motor angle to stabilize a ball around the center of the board. Note that control tasks with a given input are particularly difficult because the policy must not only react to changes in the ball's position but also learn to act in advance. The cost function is defined as:

[0223] c(x t )=1-exp(-g t (x t )),in

[0224]

[0225] The test lasted 3 seconds, with a sampling frequency of 30 Hz. The measurements provided by the camera were very noisy and could not be directly used to estimate velocity from position. We used a Kalman smoother to adjust the ball's position. and related speed Offline filtering is performed. Instead of a direct oscillation, we use a Kalman filter in the control loop to estimate the ball's state online based on noise measurements of its position. When simulating the online estimator during policy optimization, we attempt to predict the particle's position with and without some additive noise perturbation. We obtain similar performance in both cases, likely due to the Kalman filter's ability to effectively filter out the white noise added to the particle.

[0226] Regarding the model, we only need to learn two GPs that predict the evolution of the ball's velocity, because we directly control the motor angles; therefore, we assume their evolution is deterministic. GP inputs Including status

[0227] An extended version, in which the angle has been replaced by its sine and cosine, and has utilized causal numerical differentiation (T) s The motor angular velocity was estimated using the sampling time. SE+P was used. (1) Kernel (10), where the linear kernel acts only on a subset of the model input. The GP reduction threshold is reduced to 10 because the ball can cover a small distance in one time step. -4 We considered M=400 particles for policy gradient estimation. The policy unit is a particle with n... b =400 basis functions in the multi-output RBF-network(11), which receives the basis functions calculated using a Kalman filter. The estimate is used as input; the maximum angular displacement of the two motors is u. max =4 [deg]. The optimization parameters used in the strategy are... Figure 1E The description is the same as in [the previous text], the difference being that we use α. lr =0.006 as the initial learning rate. The reduction in the learning rate is related to the use of small-scale intervals in the cost function, which is necessary to handle small movements of the ball. For the same reason, we also set... and σ s=0.05. The initial exploration is given by two different trials, where the control signals are two triangular waves perturbed by white noise. In most cases, during the exploration and initial trials, the ball may touch the boundary of the board. In these cases, we hold the data until the moment of impact. A characteristic of this experiment compared to other experiments seen before is the wide range of initial conditions. In fact, the ball can be positioned anywhere on the surface of the board, and the strategy must control it to the center. and The initial distribution is a uniform U(-0.15, 0.15), which almost covers the entire surface (the plate is a square with sides of approximately 0.20 [m]). For other state components... and Furthermore, we assume a more compact initial distribution U(-10) -6 10 -6 The MC-PILCO4PMS manages to learn a strategy to control the ball around the center of gravity from any initial position after the third trial, which involves 11.33 seconds of interaction with the system. See [link to relevant documentation]. Figure 14 .

[0228] According to some embodiments of the invention, the proposed framework can use GP to derive a probabilistic model of the system dynamics and update the policy parameters through gradient-based optimization; this optimization utilizes reparameterization techniques and relies on Monte Carlo methods to approximate the expected cumulative cost. Compared to similar algorithms proposed in the past, the Monte Carlo method focuses on two aspects: (i) the appropriate choice of the cost function, and (ii) the introduction of exploration during policy optimization through the use of dropout techniques. We compare MC-PILCO with PILCO and Black-DROPS, two existing GP-based MBRL algorithms. MC-PILCO outperforms both algorithms, exhibiting better data efficiency and asymptotic performance. The results obtained in simulations confirm the effectiveness of the proposed solution and demonstrate the importance of the above two aspects when optimizing a policy that combines reparameterization techniques with particle-based methods. Furthermore, due to the particle-based approximation associated with moment matching employed in PILCO, we explore two advantages: the possibility of using structured kernels such as polynomial kernels and semi-parametric kernels, and the ability to handle multimodal distributions. Specifically, results obtained in simulations and using real systems show that the use of structured kernels can improve data efficiency and reduce the interaction time required for learning tasks. Some implementations illustrate systems with partially measurable states, which are particularly meaningful in practical applications. Furthermore, some implementations can provide a modified algorithm called MC-PILCO4PMS, where we validate the importance of considering the state estimator used in real systems during policy optimization. Results are shown in different simulation scenarios (i.e., a cart-like inverted pendulum and a robot manipulator), and also in real systems such as the Furuta pendulum and ball-and-disc setups.

[0229] The embodiments of the present invention described above can be implemented in any of a variety of ways. For example, the embodiments can be implemented in hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or group of processors, whether the software code is provided in a single computer or distributed among multiple computers. Such a processor can be implemented as an integrated circuit having one or more processors within its components. However, the processor can be implemented using circuitry in any suitable manner.

[0230] Furthermore, embodiments of the present invention can be specifically implemented as methods, and examples of these methods have been provided. The actions performed as part of this method can be ordered in any suitable manner. Therefore, embodiments can be constructed that perform actions in a different order than those shown, which may include performing simultaneously some actions shown in the illustrative embodiments in sequence.

[0231] The use of ordinal terms such as "first" and "second" in the claims does not in itself imply any priority, precedence, or order of a claim element over another or temporary order of the actions of the method, but is merely used as a marker to distinguish one claim element with a specific name from another element with the same name (but using ordinal terms) to differentiate these claim elements.

[0232] Although the invention has been described with reference to preferred embodiments, it will be understood that various other changes and modifications can be made within the spirit and scope of the invention.

[0233] Therefore, the purpose of the appended claims is to cover all such variations and modifications that fall within the true spirit and scope of the invention.

Claims

1. A controller (100) for a control system (11), the system including a strategy configured to control the system (11), the controller comprising: An interface connected to the system (11), the interface being configured to acquire action state and measurement state via sensors measuring the system (11); The memory is used to store computer-executable program modules including model learning modules (1300, 132, 1300B, 1300C) and policy learning modules (1400, 1400B, 1400C); Processor (120), the processor being configured to perform the steps of the program module, the steps including: Offline modeling is performed to generate offline states based on the action states and measurement states using the model learning module. The model learning module (1300, 132, 1300B, 1300C) includes an offline state estimator (131) and a model learning program configured to model the system using a kernel function or semi-parametric kernel composed of a combination of squared exponential kernels and multiplicative polynomial kernels. The offline state estimator (131) estimates the offline states and provides them to the model learning program, which takes the offline states as input and learns a system model simulating the real system, providing the system model to the policy learning module (1400, 1400B, 1400C). The policy learning modules (1400, 1400B, 1400C) generate Monte Carlo (MC)-based particles, which are obtained by sampling from a probability distribution using the Monte Carlo method to estimate the expected cumulative cost of the particle trajectory propagating through the learned model. The policy learning modules (1400, 1400B, 1400C) include an online state estimator model (143) configured to generate an online particle estimate based on particle measurements and previous online particle estimates. The model learning module provides the policy learning module (1400, 1400B, 1400C) with the system model learned based on the offline state to generate policy parameters for MC-based particles; and The policy of the system (11) is updated based on the updated policy parameters to operate the system (11).

2. The controller (100) according to claim 1, wherein, The model learning module (1300, 132, 1300B, 1300C) uses a velocity integral model to learn the behavior of the real system, and / or wherein the model learning module (1300, 132, 1300B, 1300C) generates an offline learning state and provides it to the policy learning module (1400, 1400B, 1400C), and / or wherein the policy learning module (1400, 1400B, 1400C) includes a policy optimization program, wherein the policy optimization program performs policy optimization based on the offline state from the model learning module (1300, 132, 1300B, 1300C) and generates the policy parameters.

3. The controller (100) according to claim 2, wherein, When the policy learning module (1400, 1400B, 1400C) includes a policy optimization program, the policy optimization program performs policy optimization and generates the policy parameters based on the offline state from the model learning module (1300, 132, 1300B, 1300C). The policy learning module (1400, 1400B, 1400C) includes a system model, wherein the system model generates particle states based on previous particle states and the action states.

4. The controller (100) according to claim 3, wherein, The policy learning module (1400, 1400B, 1400C) includes a sensor model configured to generate particle measurements based on the particle state.

5. The controller (100) according to claim 1, wherein, The policy learning module (1400, 1400B, 1400C) includes a policy optimization unit configured to generate policy parameters based on particle measurements and online particle estimates, wherein the policy optimization unit provides the policy parameters to update the policy unit of the real system, and / or wherein the policy optimization unit includes a drop method and an early stopping policy, the early stopping policy being configured to improve the policy parameters generated by the policy optimization unit, and / or wherein the offline state estimator (131) is formed by a noncausal filter, a Kalman smoother, or a central difference velocity approximator.

6. A vehicle control system (100B) for controlling the movement of a vehicle, said vehicle control system include: According to claim 1, the controller (100) is connected to the motion controller (150B) of the vehicle and a vehicle motion sensor (1101) for measuring the motion of the vehicle, wherein the control system generates strategy parameters based on the motion measurement data, and wherein the control system provides the strategy parameters to the motion controller (150B) of the vehicle to update the strategy unit of the motion controller (150B).

7. The vehicle control system (100B) according to claim 6, wherein, The motion controller (150B) is configured to control the suspension of the vehicle, and / or wherein the motion controller (150B) is configured to control the actuators of the vehicle.

8. The vehicle control system (100B) according to claim 6, wherein, The model learning module is configured to generate an offline learning state and provide it to the policy learning module (1400, 1400B, 1400C), wherein the policy learning module (1400, 1400B, 1400C) generates policy parameters.

9. The vehicle control system (100B) according to claim 8, wherein, The policy learning module (1400, 1400B, 1400C) includes an online state estimator (13), wherein the online state estimator (13) performs policy optimization and generates the policy parameters based on the offline learning state of the model learning module (1300, 132, 1300B, 1300C).

10. The vehicle control system (100B) according to claim 8, wherein, The policy learning module (1400, 1400B, 1400C) includes a system model and a sensor model, wherein the system model generates particle measurement values ​​based on particle states.

11. The vehicle control system (100B) according to claim 10, wherein, The policy learning modules (1400, 1400B, 1400C) include online state estimator models configured to generate online particle estimates based on the particle measurements and previous online particle measurements.

12. A robot control system for controlling the motion of a robot, the robot control system comprising: According to claim 1, the controller (100) is connected to the actuator controller (150C) of the robot and a sensor configured to measure the state of the robot, wherein the control system generates policy parameters based on the measurement data of the sensor, and wherein the control system provides the policy parameters to the actuator controller (150C) of the robot to update the policy unit of the actuator controller (150C).

13. The robot control system according to claim 12, wherein, The actuator controller (150C) is configured to control at least one actuator of the robot, and / or wherein the actuator controller (150C) is configured to control more than one actuator of the robot, and / or wherein the model learning module (1300, 132, 1300B, 1300C) in the controller (100) includes an offline state estimator (131) and a model learning module, wherein the offline state estimator (131) estimates the offline state and provides it to the model learning module.

14. The robot control system according to claim 13, wherein, When the model learning modules (1300, 132, 1300B, 1300C) in the controller (100) include an offline state estimator (131) and a model learning module, wherein the offline state estimator (131) estimates the offline state and provides it to the model learning module, and the model learning module is configured to generate an offline learning state and provide it to the policy learning modules (1400, 1400B, 1400C), wherein the policy learning modules (1400, 1400B, 1400C) 00C) generates policy parameters, and / or when the model learning module (1300, 132, 1300B, 1300C) in the controller (100) includes an offline state estimator (131) and a model learning module, wherein the offline state estimator (131) estimates the offline state and provides it to the model learning module, and the policy learning module (1400, 1400B, 1400C) includes a system model and a sensor model, wherein the system model generates particle measurements based on particle states.

15. The robot control system according to claim 14, wherein, The policy learning module (1400, 1400B, 1400C) includes an online state estimator (13), wherein the online state estimator (13) performs policy optimization and generates the policy parameters based on the offline learning state of the model learning module (1300, 132, 1300B, 1300C), and / or wherein the policy learning module (1400, 1400B, 1400C) includes an online state estimator model configured to generate an online particle estimate based on the particle measurement and previous online particle measurement.

Images