Systems and methods for controlling the operation of a device

Abstract

The present disclosure provides a feedback controller and method for controlling the operation of a device at different control steps. The feedback controller includes at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the feedback controller to collect a measurement indicating the state of the device at a control step for the control step and recursively execute a parameterized probabilistic solver on a control input to an actuator that operates the device until an end condition is met to generate a control input for the control step. The feedback controller is further configured to control an actuator that operates the device based on the generated control input.

Description

Technical Field

[0001] The present disclosure generally relates to control systems, and more particularly, to systems and methods for controlling the operation of a device in different control steps based on feedback signals.

Background Art

[0002] Optimal control addresses finding control for a dynamic system over a period of time such that an objective function is optimized. This has numerous applications in science, engineering, and operations research. For example, the dynamic system may be a spacecraft with control corresponding to rocket thrusters, and the objective may be to reach the moon with minimal fuel consumption. Similarly, the dynamic system may be a vehicle with control corresponding to vehicle acceleration. Model-based control techniques such as linear quadratic regulator (LQR) or model predictive control (MPC) use a mathematical model of the dynamic system to obtain actuator commands / inputs. For numerical reasons, such dynamic system models are simplified to facilitate numerical optimization, i.e., it is a "control-oriented" model. Such control-oriented models are selected to be analytical functions that are linear / nonlinear and continuous, and thus are suitable for gradient-based optimization.

[0003] As control applications become increasingly complex and computational resources become more powerful, the interest in and the ability of more advanced controllers to handle complex behavior have increased to cope with complex dynamics. For example, to accurately control a dynamic system, it may be necessary to consider behaviors that are difficult to analytically model as explicit functions. Such behaviors include contact mechanics, friction, inertia of complex shapes, flexible bodies such as in soft robotics, implicit differential equations, and the like. The formulation of optimal control problems that take complex dynamics into account poses several computational challenges that are difficult to address in real-time control applications. For example, dynamic programming used to find an optimal solution to an optimal control problem can fail when the dynamic model includes discontinuous functions.

[0004] Accordingly, there is a need for systems and methods for solving optimal control problems suitable for controlling systems with complex dynamics. SUMMARY OF THE INVENTION

[0005] An object of some embodiments is to provide a system and method suitable for feedback control of a system with complex dynamics. Additionally, or alternatively, an object of some embodiments is to provide feedback control for a system / device whose dynamics have an unknown analytical form. Examples of such devices include legged robots, robots or systems partially made of compliant materials instead of rigid links, electric motors, and the like. In fact, friction in the operation of many robotic systems can be difficult to capture in an analytical form.

[0006] Some embodiments are based on the understanding that, to achieve such an object, an alternative approach for solving optimal control problems that is different from searching for analytical solutions and / or iterative optimizations, for example, different from gradient-based optimization, is required.

[0007] An example of such an alternative approach for solving complex optimization problems is based on the principle of training a deep neural network (DNN) starting from a new technical field of physics-informed neural networks (PINN). The training of a DNN is performed by minimizing a loss function so as to build a model based on training data to make predictions or judgments without being explicitly programmed to make predictions or judgments. Therefore, the use of a DNN is generally divided into two stages, namely, a training stage for learning the parameters of the DNN and a testing stage for making judgments using the trained DNN. Therefore, the output generated by the DNN during the training stage is typically ignored since it is only used to train the parameters of the DNN.

[0008] However, training a DNN to minimize a loss function can be envisioned as finding a solution to the loss function or, specifically, finding the parameters of the DNN that force the trained DNN to output the solution of the loss function given a certain input. Some embodiments are based on the understanding that the loss function can be formulated as an optimal control problem. In this regard, training a neural network using a loss function can be envisioned as being equivalent to solving an optimal control problem. For example, the output of a trained neural network can be the control input for the current control step to the device to be controlled. The input to the neural network can include a feedback signal indicating the state of the device to be controlled at the current control step. Therefore, the neural network needs to be retrained for each control step.

[0009] However, while this example provides an alternative approach to solving the optimal control problem, formulating the optimal control problem as training a DNN suffers from slow convergence and is not very suitable for real-time control applications. The objective of some embodiments is to find different alternative approaches for solving the optimal control problem.

[0010] Some embodiments are based on the recognition that the reason for the slow convergence of DNN training posed to solve the optimal control problem lies in the deterministic nature of the output of the DNN. Although the DNN can be interpreted as a multivariate statistical model for approximating an unknown expected function, the output generated by the output layer of the DNN is deterministic and finite without further modification. In addition, another reason for the slow convergence can be found in the deterministic update of the parameters of the DNN. This determinism slows down the backpropagation training of the DNN that explores the parameters of the DNN to bring about the minimization of the loss function.

[0011] Some embodiments are based on the recognition that the estimation of the solution to the optimal control problem and the iterative search for such a solution should be probabilistic. The probabilistic nature of the solution estimation and the probabilistic update of the current solution can increase the convergence of the estimated solution to the optimal solution, because it enables exploring the solution space with likely optimal solutions.

[0012] To that end, the objective of some embodiments is to find a probabilistic framework that enables probabilistically exploring the control inputs for controlling the device provided by the solution to the optimal control problem. Further, the objective of some embodiments is to find such a probabilistic framework that probabilistically updates the current probabilistic estimation of the control inputs until the termination condition is met.

[0013] Examples of such probabilistic frameworks explored by various embodiments are probabilistic solvers such as the Kalman filter. Probabilistic solvers use a series of measurements observed over time, including statistical noise and other inaccuracies, to generate estimates of unknown variables that tend to be more accurate than those based on only a single measurement. Probabilistic solvers are used to track the state of a device under control. In other words, the Kalman filter is parameterized with respect to the state of the device under control. For example, the Kalman filter can be used to estimate the state of a vehicle from measurements of satellite signals, as in a GNSS application example. In this example, the Kalman filter is parameterized with respect to the state of the vehicle.

[0014] The framework of the operation of a probabilistic solver includes two phases: a prediction phase and a correction phase. For the prediction phase, the probabilistic solver generates estimates of the current state variables, along with their uncertainties. To do so, the probabilistic solver uses a prediction model that is affected by process noise. An example of a prediction model is a motion model of the device under control, such as a vehicle's motion model. When the result of the next measurement (necessarily corrupted with some error including random noise) is observed, the estimate is updated using a weighted average of the measurements, with more weight given to measurements with greater certainty. To do so, the probabilistic solver uses a measurement model that is affected by measurement noise. The measurement model connects the measurements to the state estimate. For example, the measurement model can connect GNSS measurements to the state of a vehicle.

[0015] Both process noise and measurement noise can be represented by a probability density function (PDF) that indicates the likelihood of variability of the predicted state and / or received measurements. Probabilistic solvers are recursive and can operate in real time using only the current input measurements and previously estimated states and their uncertainty matrices; additional past information is not required but can be used if desired.

[0016] Some embodiments are based on the principle that a Kalman filter can provide a framework for searching for a solution to an optimal control problem in a probabilistic manner. In fact, instead of having a Kalman filter parameterized with respect to state variables such as the state of a vehicle, the Kalman filter can be parameterized with respect to control inputs such as the acceleration value of the vehicle. Thus, the prediction model of the Kalman filter should predict the value of the control input affected by process noise in order to generate the PDF of the predicted value of the control input. An example of such a prediction model is the identity model. Another example is to make such a prediction based on a control-oriented model.

[0017] The measurement model of such a modified Kalman filter parameterized with respect to the control input should connect the measured value to the current estimated value of the control input. Further, such a connection should indicate a mismatch between the control input and the estimated control input derived from the measurement, in the context of the solution to the optimal control problem.

[0018] Some embodiments are based on the recognition that the cost function of an optimal control problem can be evaluated through simulation of the operation of the device using the predicted control input and the current measured value of the operation of the device. For example, the simulation can be performed based on a digital twin of the device under control to estimate a metric of the performance of the operation of the device with the control input being evaluated. The result of the simulation is evaluated with respect to the cost function and mapped in the control space affected by measurement noise. For that purpose, the measurement model of some embodiments connects a metric of the performance of the operation of the device to the control input affected by measurement noise to estimate the PDF of the measured value of the control input.

[0019] Furthermore, the correction step of the Kalman filter updates the PDF of the predicted value of the control input based on the PDF of the measured value of the control input to generate the PDF of the value of the control input for subsequent iterations. In this way, the control input is iteratively estimated and updated in a probabilistic manner. This iteration is repeated multiple times for each control step until the termination condition is satisfied.

[0020] Accordingly, one embodiment discloses a feedback controller for controlling the operation of a device in different control steps based on a feedback signal including measurement values indicating the state of the device in the different control steps. The feedback controller includes at least one processor and a memory storing instructions which, when executed by the at least one processor, cause the feedback controller to collect, for a control step, measurement values indicating the state of the device in the control step, recursively until an end condition is met, execute a parameterized probabilistic solver on a control input to an actuator that operates the device to generate a control input for the control step, and during each such execution, the probabilistic solver is configured to estimate a probability density function (PDF) of a predicted value of the control input from a PDF of values of the control input using a prediction model, evaluate a cost function of an optimal control problem for controlling the device based on a simulation of the operation of the device with the measurement values and a value of the control input sampled from the PDF of the predicted value of the control input to generate a performance metric of the operation of the device, estimate a PDF of a simulated value connected to the control input based on a measurement model connecting the performance metric of the operation of the device to the control input, and correct the PDF of the predicted value of the control input based on the PDF of the simulated value connected to the control input to generate a PDF of the value of the control input. The feedback controller is further configured to control the actuator that operates the device using at least an average of the PDF of the value of the control input.

[0021] Accordingly, another embodiment discloses a method for controlling the operation of a device in different control steps based on a feedback signal including measured values indicating the state of the device in the different control steps. The method includes collecting measured values indicating the state of the device in the control step, and recursively executing a parameterized probabilistic solver on a control input to an actuator that operates the device until an end condition is met to generate a control input for the control step. During each such execution, the probabilistic solver is configured to estimate a probability density function (PDF) of a predicted value of the control input from a PDF of values of the control input using a prediction model, and evaluate a cost function of an optimal control problem for controlling the device based on at least one simulation of the operation of the device with the measured value and a value of the control input sampled from the PDF of the predicted value of the control input to generate a performance metric of the operation of the device, configure to estimate a PDF of a simulated value connected to the control input based on a measurement model connecting the performance metric of the operation of the device to the control input, and configure to correct the PDF of the predicted value of the control input based on the PDF of the simulated value connected to the control input to generate a PDF of the value of the control input. The method further includes controlling the actuator that operates the device using at least an average of the PDF of the value of the control input.

[0022] Accordingly, yet another embodiment discloses a non-transitory computer-readable storage medium embodying a program executable by a processor for implementing a method for controlling the operation of a device in different control steps based on a feedback signal including measurement values indicating the state of the device in the different control steps. The method includes collecting measurement values indicating the state of the device in the control step, and recursively executing a parameterized probabilistic solver on a control input to an actuator that operates the device until an end condition is met to generate a control input for the control step. During each such execution, the probabilistic solver is configured to estimate a probability density function (PDF) of a predicted value of the control input from a PDF of values of the control input using a prediction model, evaluate a cost function of an optimal control problem for controlling the device based on a simulation of the operation of the device with the measurement values and a value of the control input sampled from the PDF of the predicted value of the control input to generate a performance metric of the operation of the device, configure to estimate a PDF of a simulated value connected to the control input based on a measurement model connecting the performance metric of the operation of the device to the control input, and configure to correct the PDF of the predicted value of the control input based on the PDF of the simulated value connected to the control input to generate a PDF of the value of the control input. The method further includes controlling the actuator that operates the device using at least an average of the PDF of the value of the control input.

[0023] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The drawings shown are not necessarily to scale, and instead, generally focus on explaining the principles of the embodiments of the present disclosure.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

DETAILED DESCRIPTION OF THE INVENTION

[0025] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent to one of ordinary skill in the art, however, that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

[0026] As used in this specification and the appended claims, the phrases “for example,” “such as,” “including,” as well as the verbs “comprise,” “have,” “include,” and other verb forms, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended, meaning that the list is not to be considered as excluding other additional components or items. The phrase “based on” means at least in part based on. Further, it should be understood that the expressions and terms used herein are for the purpose of description and should not be regarded as limiting. Any headings utilized within this description are for convenience only and have no legal or limiting effect.

[0027] The object of some embodiments is to provide a system and method suitable for feedback control of a system with complex dynamics. Additionally, or alternatively, the object of some embodiments is to provide feedback control for a system / device having an analytical form with unknown dynamics. Some embodiments are based on the understanding that, in order to achieve such an object, an alternative approach for solving the optimal control problem is required, which is different from, for example, gradient-based optimization and different from the search for analytical solutions and / or iterative optimization.

[0028] Examples of such alternative approaches for solving complex optimization problems are based on the principle of training a deep neural network (DNN) starting from a new technical field of physics-informed neural networks (PINN). The training of a DNN is performed by minimizing a loss function so as to build a model based on training data to make predictions or judgments without being explicitly programmed to make predictions or judgments. Therefore, the use of a DNN is generally divided into two stages, namely, a training stage for learning the parameters of the DNN and a testing stage for making judgments using the trained DNN. Therefore, the output generated by the DNN during the training stage is typically ignored since it is only used to train the parameters of the DNN.

[0029] However, training a DNN to minimize a loss function can be envisioned as finding a solution to the loss function or, specifically, finding the parameters of the DNN that force the trained DNN to output a solution to the loss function given a certain input. Some embodiments are based on the understanding that the loss function can be formulated as an optimal control problem. In this regard, training a neural network using a loss function can be envisioned as being equivalent to solving an optimal control problem. For example, the output of a trained neural network can be the control input for the current control step to the device to be controlled. The input to the neural network can include a feedback signal indicating the state of the device to be controlled at the current control step. Therefore, the neural network needs to be retrained for each control step.

[0030] However, while this example provides an alternative approach for solving the optimal control problem, formulating the optimal control problem as training a DNN is plagued by slow convergence and is not very suitable for real-time control applications. The objective of some embodiments is to find different alternative approaches for solving the optimal control problem.

[0031] Some embodiments are based on the recognition that the reason for the slow convergence of DNN training posed to solve the optimal control problem lies in the deterministic nature of the output of the DNN. In fact, although a DNN can be interpreted as a multivariate statistical model for approximating an unknown expected function, the output generated by the output layer of the DNN is deterministic and finite without further modification. In addition, another reason for the slow convergence can be found in the deterministic update of the parameters of the DNN. This determinism slows down the backpropagation training of the DNN that explores the parameters of the DNN to bring about minimization of the loss function.

[0032] Some embodiments are based on the recognition that the estimation of the solution to the optimal control problem and the iterative search for such a solution should be probabilistic. The probabilistic nature of the solution estimation and the probabilistic update of the current solution can increase the convergence of the estimated solution to the optimal solution, because it enables exploring the solution space with likely optimal solutions.

[0033] To that end, the objective of some embodiments is to find a probabilistic framework that enables probabilistically exploring the control inputs for controlling the device provided by the solution to the optimal control problem. Further, the objective of some embodiments is to find such a probabilistic framework that probabilistically updates the current probabilistic estimate of the control input until the termination condition is met.

[0034] Examples of such probabilistic frameworks explored by various embodiments are probabilistic solvers such as the Kalman filter. Some embodiments are based on the recognition that a probabilistic solver can be used to control a device having complex dynamics. For example, based on a probabilistic solver, a feedback controller can be formulated such that it collects measurements indicating the state of the device at a control step, executes the probabilistic solver to generate a control input for that control step, and controls the device based on the generated control input. Such a feedback controller based on a probabilistic solver is described below with respect to FIG. 1.

[0035] FIG. 1 shows a block diagram of a feedback controller 100 for controlling the operation of a device 110 at different control steps, according to some embodiments of the present disclosure. The feedback controller 100 can be operably coupled to the device 110. Examples of the device 110 can include a vehicle (e.g., an autonomous vehicle), a robotic assembly, a legged robot, a motor, an elevator door, an HVAC (heating, ventilation, and air conditioning) system, etc. For example, the vehicle can be an autonomous car, an aircraft, a spacecraft, a dynamically positioned ship, etc. Examples of the operation of the device 110 can include, but are not limited to, operating the vehicle according to a specific purpose, operating the HVAC system according to specific parameters, operating a robotic arm according to a specific task, and opening and closing an elevator door.

[0036] The feedback controller 100 may include at least one processor 120, a transceiver 130, and a bus 140. Further, the feedback controller 100 may include a memory 150. The memory may be implemented as a storage medium such as RAM (Random Access Memory), ROM (Read Only Memory), a hard disk, or any combination thereof. For example, the memory 150 can store instructions executable by at least one processor 120. In certain embodiments, the memory 150 is configured to store a probabilistic solver 160 and a simulation model 170 of the device 110. The probabilistic solver 160 is parameterized with respect to a control input to an actuator that operates the device 110. The simulation model 170 of the device 110 can approximate the physical behavior of the device 110. For example, the device 110 may be an electric motor, and the simulation model 170 may be a model of the electric motor. The probabilistic solver 160 and the simulation model 170 will be described in detail at a later stage. The at least one processor 120 may be embodied as a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The at least one processor 120 may be operably connected to the memory 150 and / or the transceiver 130 via the bus 140.

[0037] According to one embodiment, the feedback controller 100 may be configured to request a sequence of control inputs to control the device 110. For example, the control input may, in some cases, be associated with physical quantities such as voltage, pressure, force, torque, etc. In an exemplary embodiment, the feedback controller 100 may request a sequence of control inputs such that the sequence of control inputs changes the state of the device 110 to perform a specific task, e.g., track a reference. When a sequence of control inputs is requested, the transceiver 130 may be configured to input the sequence of control inputs as an input signal 180 to the device 110. As a result, the state of the device 110 may be changed according to the input signal 180 to perform a specific task. For example, the transceiver 130 may be an RF (radio frequency) transceiver or the like.

[0038] Furthermore, the state of the device 110 may be measured using one or more sensors installed in the device 110. The one or more sensors can send a feedback signal 190 to the transceiver 130. The transceiver 130 may receive the feedback signal 190. In an exemplary embodiment, the feedback signal 190 may include a sequence of measurement values respectively corresponding to the sequence of control inputs. For example, the sequence of measurement values may be measurement values of the state output by the device 110 according to the sequence of control inputs. Thus, each measurement value in the sequence of measurement values can indicate the state of the device 110 caused by the corresponding control input. Each measurement value in the sequence of measurement values may, in some cases, be associated with physical quantities such as current, speed, position, etc. In this way, the feedback controller 100 may repeatedly input a sequence of control inputs and receive a feedback signal. In an exemplary embodiment, to request a sequence of control inputs for a certain control step, the feedback controller 100 uses a feedback signal 190 including a sequence of measurement values indicating the current state of the device 110.

[0039] To determine the control input for a control step, the processor 120 recursively executes the probabilistic solver 160 until the termination condition is met to generate a control input for the control step. In some embodiments, the probabilistic solver 160 may use the simulation model 170 of the device 110 to generate a control input for the control step. The steps executed by the probabilistic solver 160 to generate the control input are described below with reference to FIG. 2.

[0040] FIG. 2 shows a schematic diagram 200 illustrating the steps executed by the probabilistic solver 160 to generate a control input, according to some embodiments of the present disclosure. The steps executed by the probabilistic solver 160 include a prediction step 270, a simulation step 280, and / or a correction step 290.

[0041] In the prediction step 270, the probabilistic solver 160 uses a prediction model to estimate a predicted value probability density function (PDF) 220 of the control input from a PDF 210 of the values of the control input. FIG. 3 shows an example of the PDF 220, according to some embodiments of the present disclosure. For example, the PDF 220 may correspond to a Gaussian distribution. The Gaussian distribution can be defined by a mean 310 and a variance 320, where the mean 310 defines the central position of the distribution 220 and the variance 320 defines the spread (or width) of the distribution 220.

[0042] Referring again to FIG. 2, further, at least one simulation of the operation of the device 110 is performed using a measurement indicating the state of the device in the control step and a value of the control input sampled from the PDF 220 of the predicted value of the control input to generate a performance metric of the operation of the device 110. The probabilistic solver 160 evaluates a cost function 295 that defines a performance metric of the optimal control problem based on at least one simulation of the operation of the device 110.

[0043] Furthermore, the probabilistic solver 160 estimates the PDF 240 of the simulated values connected to the control input based on a measurement model that connects the performance metric of the device's operation to the control input. In the correction step 290, the probabilistic solver 160 may obtain the corrected PDF 250 of the predicted value of the control input using the PDF 240 of the simulated values and the PDF 230 that defines the predicted cost function 230. The correction step 290 is described in detail in FIG. 6.

[0044] The prediction step 270, the simulation step 280, and the correction step 290 are executed recursively (260) until the termination condition is satisfied. When the termination condition is satisfied, the probabilistic solver 160 outputs the value of the control input. The value of the control input defines the control policy. In other words, the value of the control input corresponds to the parameters of the control policy. The feedback controller 100 controls the device 110 according to the control policy. In particular, the feedback controller 100 controls the actuator that operates the device based on the value of the control input.

[0045] In some embodiments, the cost function 295 may include the deviation of the state of the device 110 from the state reference value, the deviation of the control input from the input reference value, a penalty for reaching a specific target late, etc. The cost function 295 may include the state of the device 110 at the current time instance provided by the current measurement value. Additionally or alternatively, the cost function 295 may include the predicted future state of the device 110. For example, the cost function 295 may use the squared 2-norm,

Number

[0046] The predicted future state x(t) of device 110 can be obtained using simulation model 170 and control policy u(t).

Number

[0047] The optimization-based procedure shown in FIG. 2 can be implemented to minimize a cost function (1) subject to the constraints defining simulation model 170 and the state of device 110 over a certain time horizon T hor

Number

[0048]

Number

[0049] In this context, the parameter θ of the control policy can define the control policy u(t), and the control policy can define the predicted future state x(t) of device 110 by the simulation model 170.

[0050] The probabilistic solver 160 can update the parameters of the control policy at every iteration i. The prediction model of the prediction step 270 can be given by the gradient of the cost function (1) with respect to the parameter θ of the control policy,

Number

Number

[0051] Alternatively, the prediction model of the prediction step 270 may be given by an identity model.

Number

[0052] The identity model is advantageous because there is no need to derive an analytical model and no need to store the analytical model in the memory 150.

[0053] Figures 4A and 4B show how the distribution of control inputs is used to determine simulation trials for the simulation step 280 according to some embodiments of the present disclosure. The distribution 410 may correspond to the PDF 220. The distribution 410 can define a specific number of simulation trials. For example, the distribution 410 can provide three control policies. The three control policies can be given by a first control input 420, a second control input 430, and a third control input 440. The probabilistic solver 160 can use the three control inputs 420, 430, and 440 to execute the simulation step 480 and evaluate the performance of the simulation. For example, the distribution 410 may be a Gaussian distribution having a mean θ and a variance σ 2 and may be.

Number

[0054] Next, using the results of the simulation trials resulting from the three control inputs, in correction step 290, the corrected PDF 250 can be obtained. For example, the control input / parameters defining the control policy can be updated using the weighted average of the three control inputs as follows.

Number

[0055] Weights w1450, w2460, and w3470 are respectively assigned to control inputs θ1420, θ2430, and θ3440. The weights w1450, w2460, and w3470 can be selected according to cost function 295,

Number

[0056] Some embodiments are based on the recognition that different variances of different Gaussian distributions can result in different control inputs being evaluated in simulation step 280. For example, a Gaussian distribution with a high variance can result in simulation trials with control inputs that are further away from the control inputs of a Gaussian distribution with a lower variance, defining the control policy.

[0057] FIG. 5 shows Gaussian distributions 510, 520, and 530 with different variances according to some embodiments of the present disclosure. The Gaussian distributions 510, 520, and 530 can be predicted by the prediction step 270. Each of the Gaussian distributions 510, 520, and 530 has a different variance with respect to each other, but the means 540 of the Gaussian distributions 510, 520, and 530 can be the same. Among the Gaussian distributions, the Gaussian distribution with the highest probability with a small variance and mean 540 can be more certain regarding the correct control input that defines the control policy. The different variances of the Gaussian distributions 510, 520, and 530 can result in different control inputs that define the control policy, which are evaluated in the simulation step 280. For example, the Gaussian distribution 530 with a high variance can result in a simulation trial with a control input that defines the control policy that is farther away from the control input of the Gaussian distribution 510 with a lower variance.

[0058] FIG. 6 is a schematic diagram showing a correction step 290 for generating a corrected PDF 250 according to some embodiments of the present disclosure. The probabilistic solver 160 corrects the PDF 220 of the predicted value of the control input based on the PDF 240 of the simulated value connected to the control input to generate the PDF 250 of the value of the control input. For example, the three control inputs 420, 430, and 440 in FIG. 4 that define three control policies and result in three predicted future states of the device 110 can be evaluated using the cost function (1). The three evaluations can be used to determine how to update the control input that defines the control policy. For example, if the evaluation of the third control input 440 results in a lower cost than the first control input 420 and a lower cost than the second control input 430, the probabilistic solver 160 can determine that the control input that defines the control policy in the next iteration should be closer to the third control input 440.

[0059] The prediction step 270, the simulation step 280, and / or the correction step 290 are recursively executed until the end condition is satisfied.

[0060] Figure 7 is a diagram showing exemplary end conditions according to some embodiments of the present disclosure. Figure 7 shows a first PDF 710, a second PDF 720, and a surface area 730 that defines the difference between the first PDF 710 and the second PDF 720. The first PDF 710 may correspond to the PDF 230 that defines the predicted cost function 230, and the second PDF 720 may correspond to the PDF 240 of the simulated values. The surface area 730 may be used as a metric for determining whether to stop the iteration or continue the recursive iteration. For example, the surface area 730 may be too large to end the recursive iteration to end the optimization-based procedure shown in Figure 2. After further iteration, the probabilistic solver 160 may generate a third PDF 740 and a fourth PDF 750. The third PDF 740 may correspond to a PDF (e.g., PDF 230) that defines the predicted cost function, and the fourth PDF 750 may correspond to the PDF 240 (e.g., PDF 240) of the simulated values. In this example, the surface area 760 between the third PDF 740 and the fourth PDF 750 may be determined to be small enough to stop the recursive iteration to end the optimization-based procedure. In some alternative embodiments, the end condition is based on a similarity metric between the PDF of the value of the control input in the current iteration and the PDF of the value of the control input in the previous iteration. If such similarity is less than a threshold, it is contemplated that the end condition is met.

[0061] For example, the optimization-based procedure may use the Kullback-Leibler information metric as an end condition. For two distributions p i+1 (θ) and p i (θ), the Kullback-Leibler information can be defined as follows.

Equation

[0062] Alternatively, the optimization-based procedure may use a metric of the proximity of the control input as follows.

Equation

[0063] Some embodiments use a Kalman filter as the probabilistic solver 160. A Kalman filter is a process (or method) that uses a series of measurements observed over a period of time, including statistical noise and other inaccuracies, to generate an estimate of an unknown variable. In fact, these generated estimates of the unknown variable (such as the control input) can be more accurate than the estimates of the unknown variable generated using a single measurement. The Kalman filter generates an estimate of the unknown variable by estimating the joint probability distribution over the unknown variable. The Kalman filter is a two-step process that includes a prediction step and an update step. In the prediction step, the Kalman filter uses a prediction model to predict the current variables along with their uncertainties governed by process noise. For example, the prediction model can be designed to receive process noise in order to reduce the uncertainty in the variables while predicting the current variables. In fact, the predicted current variables can be represented by the joint probability distribution over the current variables.

[0064] Some embodiments are based on the recognition that, because the cost function 295 and the control input are interdependent, the Kalman filter should adjust the set of parameters that define the control policy collectively. One advantage of using a Kalman filter is that the interdependence of the control inputs is taken into account by the joint distribution of the control inputs.

[0065] FIG. 8 shows an implementation of the probabilistic solver 160 as an iterative procedure using a Kalman filter 800 according to some embodiments of the present disclosure. According to an embodiment, the state of the Kalman filter 800 is defined by the control input. For this purpose, the purpose of the Kalman filter 800 is to iteratively generate the control input for different control steps. In an exemplary embodiment, the Kalman filter 800 may iteratively generate the control input using a prediction model 810 and a measurement model 840.

[0066] The prediction model 810 of the Kalman filter 800 can be used for the prediction step 270. The measurement model 840 of the Kalman filter 800 can be used as a combination of the simulation step 280 and the correction step 290. For example, the measurement model 840 of the Kalman filter 800 can be specified by the cost function 295. The cost function 295 can be interpreted as having a prior distribution given by a multivariate Gaussian distribution, [Number]

[0067] Based on the observation that maximizing the logarithm of the multivariate Gaussian distribution is equivalent to minimizing the squared 2-norm that defines the cost function 295, the probabilistic solver 160 can be used. Thus, the multivariate Gaussian distribution can define the prior distribution 850 of the cost function 295. For example, the prior distribution can be the mean h ref and the covariance Q -1 and can be given by. [Number]

[0068] To generate the control input in the current iteration, the prediction model 810 can be configured to predict the value of the control input using the prior knowledge 820 of the control input. For example, the prior knowledge 820 of the control input can be a measure of how quickly the control input is expected to change between iterations or whether it is desirable to change. For example, the control input can be expected to change according to an identity model or the gradient with respect to the cost function 295 and the process noise. The process noise can be a measure of how reliable the prediction model 810 is. The prior knowledge 820 of the control input can be a joint probability distribution (or Gaussian distribution) for the control input. The process noise can be a joint Gaussian distribution with zero mean and prior covariance P0. The process noise may be artificially designed.

[0069] The prediction model 810 can generate a predicted joint probability distribution 830 using prior knowledge 820 of the control input and the joint probability distribution for the control input in the previous iteration 860. For example, the joint probability distribution for the control input in the previous iteration 860 can be defined by the mean θ i-1|i-1 and variance (or covariance) P i-1|i-1 calculated in the previous iteration (e.g., in iteration i - 1). For example, the joint probability distribution for the control input in the previous iteration 860 can be generated based on the joint probability distribution generated in a previous past iteration (e.g., in iteration i - 2). The predicted joint probability distribution 830 for the control input can be defined by the predicted mean θ i|i-1 and the predicted variance (or covariance) P i|i-1 .

[0070] The predicted joint probability distribution 830 for the control input can be the PDF 220 of the prediction step 270 shown in FIG. 2. For example, the predicted mean θ i|i-1 and the predicted variance (or covariance) P i|i-1 can be calculated using the prediction model 810 together with an identity model,

Number

[0071] Alternatively, the predicted mean θ i|i-1 and the predicted variance (or covariance) P i|i-1 can be calculated using the prediction model 810 together with the gradient with respect to the cost function 295,

Number

[0072] It should be understood that the present disclosure is not limited to the two examples of the prediction model, and other prediction models can be used as well.

[0073] Different embodiments use different realizations of the probabilistic solver 160. Additionally or alternatively, different embodiments use different types of Kalman filters. For example, one embodiment uses the framework of an unscented Kalman filter. This embodiment is advantageous because the unscented Kalman filter performs updates without estimating gradients during its correction step. Thus, this embodiment can solve optimization problems without estimating gradients, which is beneficial for some dynamic devices with complex dynamics.

[0074] Furthermore, the unscented Kalman filter can advantageously show samples of control inputs for the simulation step 280. For example, the unscented Kalman filter represents the PDF of the values of the control inputs at sigma points and uses those sigma points through the iterations of the unscented Kalman filter to perform an unscented transform. Some embodiments are based on the recognition that the sigma points can be used as sampled control inputs to simulate its operation. Thus, the evaluation of the operation of the device is integrated into the probabilistic framework of the unscented Kalman filter.

[0075] The measurement model 840 of the Kalman filter 800 can use the predicted joint probability distribution 830 to determine simulation trials as shown in FIG. 4. Some embodiments use an unscented Kalman filter realization to determine simulation trials. The unscented Kalman filter calculates sigma points that represent the predicted joint probability distribution 830. Thus, the sigma points define an alternative definition of the predicted joint probability distribution 830. Each of the sigma points characterizes one simulation trial in the simulation step 280.

Number

[0076]

Number

[0077] Given a simulated sigma point-based state trajectory, measurement model 840 can evaluate the simulated performance for each of the sigma points using cost function 295.

Number

[0078] The weights of the unscented Kalman filter may be selected differently. For example, the weights may be selected to have equal weights.

Number

[0079] Sigma points can be calculated using the covariance matrix and Cholesky decomposition.

Number

[0080] The unscented Kalman filter is advantageous because it defines both the simulation trial to be evaluated and how to update the control input using the covariance matrix and the estimated value of the control input and the result of the simulation trial.

[0081] Measurement model 840 can execute correction step 290 using the simulated performance for each of the sigma points.

Number

[0082] Some embodiments iteratively update the control input using the sigma points and the simulated performance of the sigma points.

Number

[0083] It should be understood that the disclosed embodiments represent exemplary embodiments. In fact, other embodiments such as different weight selections, different sigma point calculations, or different prediction models can be similarly selected.

[0084] Next, the correction step 290 of the measurement model 840 can output the updated mean θ of the control input 870 i|i and the updated covariance P i|i The updated combined distribution 880 of the control input can then be used to determine the input signal 180 to the device 110 when the end conditions as shown in FIG. 7 are met. If the end conditions are not met, the next iteration 890 is executed.

[0085] Some embodiments are based on the recognition that a further advantage of having a parameterized Kalman filter for the control input is that the Kalman filter can test some samples from the PDF 220 of the predicted values to accelerate convergence. Testing multiple samples results in multiple simulations. However, the testing and simulations can be performed simultaneously using multiple parallel processors. Additionally, different samples can be tested using different Kalman filters with different measurement noises.

[0086] FIG. 9 shows an exemplary list of the purposes of the cost function 295 according to some embodiments of the present disclosure. The cost function can be any combination of elements in the list. For example, the cost function can include the cost 910 for the deviation of any number of states from the target state, h ref = x ref and h(x(t), u(t)) = x(t); the cost 915 for the deviation of any number of inputs from the target input, h ref = u ref and h(x(t), u(t)) = u(t).

[0087]

Number

[0088] Additionally or alternatively, it is possible to select the objective to include the cost for a particular control input itself (rather than a state or input), for example,

Number

[0089] In one embodiment, feedback controller 100 controls robot 1010 from initial position 1040 to target position 1050 such that the deviation from path 1060 of robot 1010 is minimized. For example, feedback controller 100 collects measurements indicating the state of robot 1010, such as the current position of robot 1010. Further, feedback controller 100 recursively executes probabilistic solver 160 until the termination condition is satisfied to generate a control input that maintains the position of robot 1010 along path 1060. Feedback controller 100 further controls robot 1010 based on the generated control input.

[0090] In addition, in some embodiments, feedback controller 100 may be used to control a crane that manipulates a load. For example, feedback controller 100 may minimize the vibration of the load of the crane with respect to a reference path while manipulating the load.

[0091] This description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with a feasible description for implementing one or more exemplary embodiments. It is contemplated that various changes can be made in the functions and configurations of the elements without departing from the spirit and scope of the subject matter disclosed as set forth in the claims.

[0092] In the following description, specific details are provided for a thorough understanding of the embodiments. However, it should be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Additionally, like reference numerals and names in the various drawings indicate like elements.

[0093] Also, individual embodiments may be described as a process shown as a flowchart, a flow diagram, a data flow diagram, a structural diagram, or a block diagram. A flowchart may describe the operations as a sequential process, but many of the operations can be executed in parallel or simultaneously. Additionally, the order of the operations may be rearranged. The process may end when its operations are completed, but may have additional steps not discussed or included in the figure. Further, not all operations in any particular process described will occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the end of the function can correspond to the return of the function to the calling function or the main function.

[0094] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, either manually or automatically. Manual or automatic implementations may be executed or at least supported using a machine, hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored on a machine-readable medium. The necessary tasks may be executed by a processor.

[0095] The various methods or processes outlined herein may be coded as software executable on one or more processors using any one of a variety of operating systems or platforms. Additionally, such software may be written using any of several suitable programming languages and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine. Typically, the functionality of program modules may be combined or distributed as desired in various embodiments.

[0096] Embodiments of the present disclosure may be embodied as a method for which an example is provided. The acts performed as part of the method may be ordered in any suitable manner. Accordingly, embodiments may be constructed in which acts shown as consecutive acts in exemplary embodiments are performed simultaneously, including the performance of some acts in a different order than that illustrated.

[0097] Furthermore, the embodiments of the present disclosure and the functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Additionally, some embodiments of the present disclosure can be realized as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, a data processing apparatus. Further, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.

[0098] According to an embodiment of the present disclosure, the term "data processing apparatus" can include, by way of example, all kinds of devices, apparatuses, and machines for processing data, including programmable processors, computers, or multiple processors or computers. The apparatus can include dedicated logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The apparatus can also include, in addition to hardware, code that generates an execution environment for the computer program, e.g., code constituting processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0099] Computer programs (which may also be referred to as or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, either as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program may or may not correspond to a file in a file system. The program can be stored as part of a file that holds other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple cooperating files, such as files that hold one or more modules, subprograms, or portions of code.

[0100] A computer program can be deployed to execute on one computer, or located on one site, or distributed across multiple sites and executed on multiple computers interconnected by a communication network. Computers suitable for the execution of a computer program include, by way of example, general-purpose microprocessors or special-purpose microprocessors or both, and any other kind of central processing unit, and may be based thereon, for example. Generally, the central processing unit receives instructions and data from read-only memory or random access memory or both. Essential elements of a computer are a central processing unit for executing instructions, and one or more memory devices for storing instructions and data.

[0101] Generally, a computer will also be operatively coupled to include, or receive data from, or transfer data to, or both, one or more mass storage devices for storing data, such as magnetic disks, magneto - optical disks, or optical disks. However, a computer need not have such devices. Further, a computer can be incorporated into other devices, such as a cellular phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable memory device, such as a universal serial bus (USB) flash drive, to name a few.

[0102] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and a pointing device, such as a mouse or a trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; input received from the user can be received in any form including acoustic input, speech (voice) input, or tactile input. Further, the computer can interact with the user by sending documents to and receiving documents from the devices used by the user, such as by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0103] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes, for example, backend components as a data server, or includes middleware components such as, for example, an application server, or a frontend component, such as a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication in any form or medium, such as by a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0104] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between a client and a server arises by computer programs running on respective computers and having a client-server relationship to each other.

[0105] Although the present disclosure has been described with reference to particular preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the present disclosure. Accordingly, it is an aspect of the claims to embrace all such variations and modifications that fall within the true spirit and scope of the present disclosure.

Claims

1. A feedback controller for controlling the operation of a device in different control steps based on a feedback signal including measurement values indicating the state of the device in the different control steps, comprising at least one processor and a memory storing instructions, which, when executed by the at least one processor, cause the feedback controller to, for a control step, collect measurement values indicating the state of the device in the control step, recursively until an end condition is met, execute a parameterized probabilistic solver on a control input to an actuator that operates the device to generate a control input for the control step, and during each execution, the probabilistic solver is configured to estimate a probability density function (PDF) of a predicted value of the control input from a PDF of values of the control input using a prediction model, evaluate a cost function of an optimal control problem for controlling the device based on a simulation of the operation of the device with the measurement values and values of the control input sampled from the PDF of the predicted value of the control input to generate a performance metric of the operation of the device, estimate a PDF of a simulated value connected to the control input based on a measurement model connecting the performance metric of the operation of the device to the control input, correct the PDF of the predicted value of the control input based on the PDF of the simulated value connected to the control input to generate a PDF of the value of the control input, and the instructions, when further executed by the at least one processor, cause the feedback controller to, for the control step, control the actuator that operates the device using at least an average of the PDF of the value of the control input, a feedback controller.

2. The feedback controller according to claim 1, wherein the prediction model is an identity model.

3. The feedback controller according to claim 1, wherein the prediction model is given by a gradient of the cost function with respect to the control input.

4. For each iteration executed for each of the control steps, the probabilistic solver is further configured to evaluate the cost function for a plurality of samples defining simulation trials of the PDF of the predicted value, the feedback controller of claim 1.

5. From a plurality of performance metrics, to generate the PDF of the simulated value connected to the control input, the probabilistic solver is further configured to determine a weighted combination of evaluations of a plurality of samples of the PDF of the predicted value, the feedback controller of claim 4.

6. The cost function for a plurality of samples of the PDF of the predicted value is evaluated in parallel using a plurality of processors, the feedback controller of claim 4.

7. The termination condition is based on a similarity metric between the PDF of the value of the control input and the PDF of the value of the control input in the previous iteration, the feedback controller of claim 1.

8. The probabilistic solver is a Kalman filter, the feedback controller of claim 1.

9. The probabilistic solver is an ensemble Kalman filter that performs gradient-free correction of the predicted value of the control input, the feedback controller of claim 1.

10. To update the PDF of the predicted value using the gradient-free correction, the ensemble Kalman filter is configured to generate the PDF of the simulated value connected to the control input using the evaluation of the cost function, the feedback controller of claim 9.

11. The ensemble Kalman filter evaluates the cost function for a plurality of sigma points of the PDF of the predicted value, the feedback controller of claim 9.

12. To determine the sigma points, the ensemble Kalman filter is configured to generate the sigma points based on the predicted mean of the PDF of the predicted value and the covariance matrix of the PDF of the predicted value, the feedback controller of claim 11.

13. The sigma points are determined based on Cholesky decomposition, the feedback controller of claim 12.

14. A method for controlling the operation of a device in different control steps based on a feedback signal including measurement values indicating the state of the device in the different control steps, comprising: collecting measurement values indicating the state of the device in the control step; recursively executing a parameterized probabilistic solver on a control input to an actuator that operates the device until an end condition is met, to generate a control input for the control step, wherein during each execution, the probabilistic solver is configured to estimate a probability density function (PDF) of a predicted value of the control input from a PDF of values of the control input using a prediction model; evaluating a cost function of an optimal control problem for controlling the device based on a simulation of the operation of the device with the measurement values and values of the control input sampled from the PDF of the predicted value of the control input, to generate a performance metric of the operation of the device; configured to estimate a PDF of a simulated value connected to the control input based on a measurement model connecting the performance metric of the operation of the device to the control input; configured to correct the PDF of the predicted value of the control input based on the PDF of the simulated value connected to the control input to generate a PDF of the value of the control input, the method further comprising controlling the actuator that operates the device using at least an average of the PDF of the value of the control input.

15. The method according to claim 14, wherein the prediction model is an identity model.

16. The method according to claim 14, wherein the prediction model is given by a gradient of the cost function with respect to the control input.

17. The method according to claim 14, wherein for each iteration executed for each of the control steps, the probabilistic solver is further configured to evaluate the cost function for a plurality of samples defining a simulation trial of the PDF of the predicted value.

18. The method according to claim 17, wherein, from a plurality of performance metrics, the probabilistic solver is further configured to determine a weighted combination of evaluations of a plurality of samples of the PDF of the predicted value to generate the PDF of the simulated value connected to the control input.

19. The method according to claim 14, wherein the end condition is based on a similarity metric between the PDF of the value of the control input and the PDF of the value of the control input in the previous iteration.

20. A non-transitory computer-readable storage medium embodying a program executable by a processor for executing a method for controlling the operation of a device in different control steps based on a feedback signal including measurement values indicating the state of the device in the different control steps, the method comprising: collecting measurement values indicating the state of the device in the control step; recursively executing a parameterized probabilistic solver on a control input to an actuator that operates the device until an end condition is met to generate a control input for the control step, wherein during each execution, the probabilistic solver is: configured to estimate a PDF of a predicted value of the control input from a probability distribution function (PDF) of the value of the control input using a prediction model; evaluating a cost function of an optimal control problem for controlling the device based on a simulation of the operation of the device involving the measurement value and the value of the control input sampled from the PDF of the predicted value of the control input to generate a performance metric of the operation of the device; configured to estimate a PDF of a simulated value connected to the control input based on the performance metric of the operation of the device and a measurement model connecting the performance metric to the control input; configured to correct the PDF of the predicted value of the control input based on the PDF of the simulated value connected to the control input to generate the PDF of the value of the control input, the method further comprising: controlling the actuator that operates the device using at least an average of the PDF of the value of the control input.

Images