Feedback controller, method, and non-temporary computer-readable storage medium

Abstract

The present disclosure provides a feedback controller and method for controlling the operation of a device at different control steps. The feedback controller includes at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the feedback controller to collect a measurement indicating the state of the device at a control step for the control step and recursively execute a parameterized probabilistic solver on a control input to an actuator that operates the device until an end condition is met to generate a control input for the control step. The feedback controller is further configured to control an actuator that operates the device based on the generated control input.

Description

[Technical Field]

[0001] This disclosure relates in general to control systems, and more particularly to systems and methods for controlling the operation of a device in different control steps based on feedback signals. [Background technology]

[0002] Optimal control deals with finding control for a dynamic system over a period of time so that an objective function is optimized. This has numerous applications in scientific, engineering, and operational research. For example, the dynamic system may be a spacecraft with control corresponding to its rocket thrusters, the objective of which may be to reach the moon with minimal fuel consumption. Similarly, the dynamic system may be a vehicle with control corresponding to its acceleration. Model-based control techniques, such as linear secondary regulators (LQRs) or model predictive control (MPCs), use mathematical models of the dynamic system to determine actuator commands / inputs. For numerical reasons, such dynamic system models are simplified to facilitate numerical optimization; i.e., they are "control-oriented" models. Such control-oriented models are chosen to be analytical functions that are linear / nonlinear and continuous, and are therefore suitable for gradient-based optimization.

[0003] As control applications become increasingly complex and computational resources become more powerful, interest in and capabilities of more advanced controllers are growing to address complex behaviors. For example, precisely controlling dynamic systems may require considering behaviors that are difficult to model analytically as explicit functions. Such behaviors include contact dynamics, friction, inertia of complex shapes, flexible bodies such as those in soft robotics, and implicit differential equations. Formulating optimal control problems that consider complex dynamics presents several computational challenges that are difficult to address in real-time control applications. For instance, dynamic programming used to find the optimal solution to an optimal control problem may fail if the dynamic model contains discontinuous functions.

[0004] Therefore, there is a need for systems and methods to solve the optimal control problem suitable for controlling systems with complex dynamics. [Overview of the Initiative]

[0005] The objective of some embodiments is to provide systems and methods suitable for feedback control of systems with complex dynamics. In addition, or alternatively, the objective of some embodiments is to provide feedback control for systems / devices whose dynamics have an unknown analytical form. Examples of such devices include legged robots, robots or systems partially fabricated from compliant materials instead of rigid links, electric motors, and the like. Indeed, friction in the operation of many robotic systems can be difficult to capture in an analytical form.

[0006] Some embodiments are based on the understanding that achieving such objectives requires alternative approaches to solving optimal control problems that differ from the search for analytical solutions and / or iterative optimizations, such as gradient-based optimizations.

[0007] An example of such an alternative approach to solving complex optimization problems is based on the principle of training deep neural networks (DNNs), beginning with the newer field of physical knowledge neural networks (PINNs). Training a DNN is done by minimizing a loss function so that it builds a model on training data to make predictions or decisions without being explicitly programmed to do so. Thus, the use of DNNs is generally divided into two stages: a training stage to learn the parameters of the DNN and a testing stage to make decisions using the trained DNN. Therefore, the output generated by the DNN during the training stage is typically ignored, as it is used only to train the parameters of the DNN.

[0008] However, training a DNN to minimize a loss function can be intended as finding a solution to the loss function, or more specifically, finding the parameters of a DNN that compel the trained DNN to output a solution to the loss function given a given input. Some embodiments are based on the understanding that the loss function can be formulated as an optimal control problem. In this regard, training a neural network using a loss function can be intended as equivalent to solving an optimal control problem. For example, the output of a trained neural network may be a control input to the controlled device for the current control step. The input to the neural network may include a feedback signal indicating the state of the controlled device at the current control step. Thus, the neural network needs to be retrained for each control step.

[0009] However, while this example provides an alternative approach to solving the optimal control problem, formulating the optimal control problem as training a DNN suffers from slow convergence and is not well-suited for real-time control applications. The objective of some embodiments is to find different alternative approaches to solving the optimal control problem.

[0010] Some embodiments are based on the recognition that the reason for the slow convergence of DNN training, which is proposed to solve optimal control problems, lies in the deterministic nature of the DNN's output. While a DNN can be interpreted as a multivariate statistical model for approximating an unknown expectation function, the output generated by the DNN's output layer is deterministic and finite without further modification. In addition, another reason for slow convergence can be found in the deterministic updating of the DNN's parameters. This determinism slows down the backpropagation training of the DNN as it searches for the DNN parameters that result in minimizing the loss function.

[0011] Some embodiments are based on the recognition that the estimation of solutions to optimal control problems and the iterative search for such solutions should be probabilistic. The probabilistic nature of solution estimation and the probabilistic updating of the current solution can increase the convergence of the estimated solution to the optimal solution because it allows for the exploration of the solution space with a likely optimal solution.

[0012] To this end, an objective of some embodiments is to find a probabilistic framework that enables the probabilistic search for control inputs to control a device provided by the solution to an optimal control problem. Furthermore, an objective of some embodiments is to find such a probabilistic framework that probabilistically updates the current probabilistic estimate of the control inputs until a termination condition is met.

[0013] An example of such a probabilistic framework, explored through various embodiments, is a probabilistic solver, such as a Kalman filter. A probabilistic solver uses a series of measurements observed over time, including statistical noise and other imperfections, to generate estimates of unknown variables that tend to be more accurate than those based on a single measurement alone. Probabilistic solvers are used to track the state of a device under control. In other words, a Kalman filter is parameterized with respect to the state of a device under control. For example, a Kalman filter can be used to estimate the state of a vehicle from measurements of satellite signals, as in a GNSS application. In this example, the Kalman filter is parameterized with respect to the state of the vehicle.

[0014] The operational framework of a stochastic solver includes two phases: a prediction phase and a correction phase. In the prediction phase, the stochastic solver generates estimates of the current state variables, along with their uncertainties. To this end, the stochastic solver uses a prediction model that is affected by process noise. An example of a prediction model is a motion model of a controlled device, such as a vehicle motion model. When the results of the next measurement (which will inevitably be impaired by some error, including random noise) are observed, the estimates are updated using a weighted average of the measurements, with more weight given to measurements with greater certainty. To this end, the stochastic solver uses a measurement model that is affected by measurement noise. The measurement model connects the measurements to the state estimates. For example, the measurement model might connect GNSS measurements to the state of a vehicle.

[0015] Both process noise and measurement noise can be represented by a probability density function (PDF) that indicates the likelihood of variation in the predicted state and / or received measurement. A probabilistic solver is recursive and can operate in real time using only the current input measurement and the previously estimated state and its uncertainty matrix; additional historical information is not required, but can be used if necessary.

[0016] Some embodiments are based on the principle that a Kalman filter can provide a framework for searching for solutions to optimal control problems in a probabilistic manner. In fact, instead of having a Kalman filter parameterized on state variables such as the state of a vehicle, the Kalman filter can be parameterized on control inputs such as the acceleration values ​​of a vehicle. Thus, the predictive model of the Kalman filter should predict the values ​​of the control inputs, which are affected by process noise, in order to generate a PDF of the predicted values ​​of the control inputs. An example of such a predictive model is an identity model. Another example is making such predictions based on a control-oriented model.

[0017] Such a modified Kalman filter measurement model, parameterized for the control input, should connect the measured values ​​with the current estimates of the control input. Furthermore, such a connection should reveal the mismatch between the control input and the estimated control input derived from the measurement, within the context of the solution to the optimal control problem.

[0018] Some embodiments are based on the understanding that the cost function of an optimal control problem can be evaluated through a simulation of the device's operation using current measurements of the predicted control input and device behavior. For example, the simulation may be performed based on a digital twin of the controlled device to estimate a performance metric of the device's operation at the control input under evaluation. The simulation results are evaluated with respect to the cost function and mapped in a control space affected by measurement noise. To this end, some embodiments of the measurement model estimate the PDF of the measured value of the control input by connecting a performance metric of the device's operation to a control input affected by measurement noise.

[0019] Furthermore, the correction step of the Kalman filter updates the PDF of the predicted value of the control input based on the PDF of the measured value of the control input to generate the PDF of the value of the control input for subsequent iterations. Thus, the control input is iteratively estimated and updated in a probabilistic manner. This iteration is repeated multiple times for each control step until the end condition is satisfied.

[0020] Accordingly, one embodiment discloses a feedback controller for controlling the operation of a device in different control steps based on a feedback signal including measurements indicating the state of the device in those different control steps. The feedback controller comprises at least one processor and a memory storing instructions, which, when executed by the at least one processor, causes the feedback controller to collect measurements indicating the state of the device in a control step, and to recursively run a parameterized stochastic solver on the control inputs to actuators that operate the device until a termination condition is met, thereby generating control inputs for the control step, wherein during each execution, the stochastic solver is configured to estimate the PDF of the predicted value of the control input from the probability distribution function (PDF) of the value of the control input using a prediction model. The feedback controller is configured to evaluate the cost function of an optimal control problem for controlling the device and generate a performance metric for the device's operation, based on a simulation of the device's operation with measured values ​​and values ​​of the control input sampled from the PDF of the predicted values ​​of the control input; to estimate the PDF of the simulated values ​​connected to the control input based on a measurement model that connects the performance metric for the device's operation to the control input; and to correct the PDF of the predicted values ​​of the control input based on the PDF of the simulated values ​​connected to the control input to generate the PDF of the values ​​of the control input. The feedback controller is further configured to control the actuator that operates the device using at least the average of the PDF of the values ​​of the control input.

[0021] Accordingly, another embodiment discloses a method for controlling the operation of a device in different control steps based on a feedback signal including measured values ​​indicating the state of the device in those different control steps. The method includes collecting measured values ​​indicating the state of the device in the control step and generating control inputs for the control step by recursively running a parameterized stochastic solver on control inputs to actuators that operate the device until a termination condition is met. During each execution, the probabilistic solver is configured to estimate the PDF of the predicted value of the control input from the probability distribution function (PDF) of the value of the control input using a prediction model; to evaluate the cost function of the optimal control problem for controlling the device based on at least one simulation of the device's operation with the measured value and the value of the control input sampled from the PDF of the predicted value of the control input to generate a performance metric of the device's operation; to estimate the PDF of the simulated value connected to the control input based on a measurement model connecting the performance metric of the device's operation to the control input; and to correct the PDF of the predicted value of the control input based on the PDF of the simulated value connected to the control input to generate the PDF of the value of the control input. The method further includes controlling the actuator that operates the device using at least the mean of the PDF of the value of the control input.

[0022] Therefore, yet another embodiment discloses a non - transient computer - readable storage medium embodying a program executable by a processor for performing a method for controlling the operation of a device in different control steps based on a feedback signal including measurement values indicating the state of the device in the different control steps. The method includes collecting measurement values indicating the state of the device in the control step, and recursively executing a parameterized probabilistic solver on a control input to an actuator that operates the device until an end condition is met to generate a control input for the control step. During each such execution, the probabilistic solver is configured to estimate a probability density function (PDF) of a predicted value of the control input from a PDF of values of the control input using a prediction model, evaluate a cost function of an optimal control problem for controlling the device based on a simulation of the operation of the device with the measurement values and the value of the control input sampled from the PDF of the predicted value of the control input to generate a performance metric of the operation of the device, estimate a PDF of a simulated value connected to the control input based on a measurement model connecting the performance metric of the operation of the device to the control input, and correct the PDF of the predicted value of the control input based on the PDF of the simulated value connected to the control input to generate a PDF of the value of the control input. The method further includes controlling the actuator that operates the device using at least an average of the PDF of the value of the control input.

[0023] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The drawings shown are not necessarily to scale; instead, emphasis is generally placed on explaining the principles of the embodiments of the present disclosure.

Brief Description of the Drawings

[0024] [Figure 1]The following are block diagrams of feedback controllers for controlling the operation of a device in different control steps, according to some embodiments of the present disclosure. [Figure 2] This is a schematic diagram illustrating the steps performed by a probabilistic solver to generate a control input, according to some embodiments of the present disclosure. [Figure 3] Examples of probability distribution functions (PDF) according to several embodiments of this disclosure are shown. [Figure 4A] The present disclosure describes a procedure for how a distribution of control inputs is used to determine a simulation trial for a simulation step, according to some embodiments of this disclosure. [Figure 4B] The present disclosure describes a procedure for how a distribution of control inputs is used to determine a simulation trial for a simulation step, according to some embodiments of this disclosure. [Figure 5] The present disclosure shows several embodiments of Gaussian distributions with different variances. [Figure 6] This is a schematic diagram illustrating correction steps for generating a corrected PDF according to some embodiments of the present disclosure. [Figure 7] This figure shows exemplary termination conditions according to some embodiments of the present disclosure. [Figure 8] This disclosure describes several embodiments of a probabilistic solver as an iterative procedure using a Kalman filter. [Figure 9] The following is an exemplary list of the purposes of the cost function according to some embodiments of this disclosure. [Figure 10] This disclosure illustrates the control of a device by a feedback controller in a given environment according to several embodiments of this disclosure. [Modes for carrying out the invention]

[0025] In the following description, for illustrative purposes and to ensure a full understanding of the disclosure, numerous specific details are provided. However, it will be apparent to those skilled in the art that the disclosure can be implemented without these specific details. In other examples, the apparatus and methods are shown only in block diagram form to avoid obscuring the disclosure.

[0026] Where used herein and in the claims, the words “for example,” “as an example,” and “etc.,” and the verbs “equip,” “have,” and “include,” and their other verb forms, when used with a list of one or more components or other items, should each be interpreted as open-ended, meaning that the list should not be considered to exclude any other additional components or items. The phrase “based on” means based at least in part. Furthermore, it should be understood that the expressions and terms used herein are for illustrative purposes only and should not be considered limiting. Any headings used within this description are for convenience only and have no legal or limiting effect.

[0034] stomach Several embodiments are based on the recognition that probabilistic solvers can be used to control devices with complex dynamics. For example, based on a probabilistic solver, a feedback controller may be formulated to collect measurements indicating the state of the device in a given control step, run a probabilistic solver to generate a control input for that control step, and control the device based on the generated control input. Such a feedback controller based on a probabilistic solver is illustrated below in Figure 1.

[0035] Figure 1 shows a block diagram of a feedback controller 100 for controlling the operation of device 110 in different control steps according to several embodiments of the present disclosure. The feedback controller 100 may be operably coupled to device 110. Examples of device 110 may include vehicles (e.g., autonomous vehicles), robot assemblies, legged robots, motors, elevator doors, HVAC (heating, ventilation, and air conditioning) systems, etc. For example, vehicles may include autonomous vehicles, aircraft, spacecraft, and dynamically positioned ships. Examples of operation of device 110 may include, but are not limited to, operating a vehicle according to a particular purpose, operating an HVAC system according to specific parameters, operating a robotic arm according to a particular task, and opening and closing an elevator door.

[0036] The feedback controller 100 may include at least one processor 120, a transceiver 130, and a bus 140. Furthermore, the feedback controller 100 may include a memory 150. The memory may be implemented as a storage medium such as RAM (random access memory), ROM (read-only memory), a hard disk, or any combination thereof. For example, the memory 150 can store instructions that can be executed by at least one processor 120. In one embodiment, the memory 150 is configured to store a probabilistic solver 160 and a simulation model 170 of the device 110. The probabilistic solver 160 is parameterized with respect to control inputs to actuators that operate the device 110. The simulation model 170 of the device 110 can approximate the physical behavior of the device 110. For example, the device 110 may be an electric motor, and the simulation model 170 may be a model of an electric motor. The probabilistic solver 160 and the simulation model 170 will be described in detail in a later stage. At least one processor 120 may be embodied as a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. At least one processor 120 may be operably connected to memory 150 and / or transceiver 130 via bus 140.

[0037] According to one embodiment, the feedback controller 100 may be configured to determine a sequence of control inputs to control the device 110. For example, the control inputs may be associated with physical quantities such as voltage, pressure, force, or torque. In an exemplary embodiment, the feedback controller 100 may determine a sequence of control inputs such that the sequence of control inputs changes the state of the device 110 to perform a specific task, such as tracking a reference. Once the sequence of control inputs is determined, the transceiver 130 may be configured to feed the sequence of control inputs to the device 110 as an input signal 180. As a result, the state of the device 110 may be changed according to the input signal 180 to perform a specific task. For example, the transceiver 130 may be an RF (radio frequency) transceiver.

[0038] Furthermore, the state of device 110 can be measured using one or more sensors installed on device 110. One or more sensors can send a feedback signal 190 to transceiver 130. Transceiver 130 can receive the feedback signal 190. In an exemplary embodiment, the feedback signal 190 may include a sequence of measurements corresponding to each sequence of control inputs. For example, the sequence of measurements may be measurements of the state output by device 110 according to the sequence of control inputs. Thus, each measurement in the sequence of measurements can indicate the state of device 110 caused by the corresponding control input. Each measurement in the sequence of measurements may, optionally, be associated with a physical quantity such as current, velocity, or position. In this way, the feedback controller 100 can iteratively apply a sequence of control inputs and receive the feedback signal. In an exemplary embodiment, to determine the sequence of control inputs for a given control step, the feedback controller 100 uses a feedback signal 190 that includes a sequence of measurements indicating the current state of device 110.

[0039] To determine the control input for a given control step, the processor 120 recursively runs the probabilistic solver 160 until a termination condition is met, generating the control input for the control step. In one embodiment, the probabilistic solver 160 may use a simulation model 170 of the device 110 to generate the control input for the control step. The steps performed by the probabilistic solver 160 to generate the control input are described below in Figure 2.

[0040] Figure 2 shows a schematic diagram 200 illustrating the steps performed by the stochastic solver 160 to generate a control input, according to some embodiments of the present disclosure. The steps performed by the stochastic solver 160 include a prediction step 270, a simulation step 280, and / or a correction step 290.

[0041] In prediction step 270, the stochastic solver 160 uses a prediction model to estimate the PDF 220 of the predicted value of the control input from the probability distribution function (PDF) 210 of the control input value. Figure 3 shows an example of PDF 220 according to some embodiments of the present disclosure. For example, PDF 220 may correspond to a Gaussian distribution. A Gaussian distribution can be defined by a mean 310 and a variance 320, where the mean 310 defines the central position of the distribution 220 and the variance 320 defines the spread (or width) of the distribution 220.

[0042] Referring again to Figure 2, furthermore, in order to generate a performance metric for the operation of device 110, at least one simulation of the operation of device 110 is performed using measurements indicating the state of the device in the control step and control input values ​​sampled from PDF220 of the predicted values ​​of the control input. Based on at least one simulation of the operation of device 110, the stochastic solver 160 evaluates the cost function 295 that defines the performance metric for the optimal control problem.

[0043] Furthermore, the probabilistic solver 160 estimates the simulated value PDF240 connected to the control input based on a measurement model that connects the performance metrics of the device's operation to the control input. In the correction step 290, the probabilistic solver 160 compares the simulated value PDF240 with the predicted cost relation Number Using the defined PDF230, a corrected PDF250 can be obtained for the predicted value of the control input. The correction step 290 is described in detail in Figure 6.

[0044] The prediction step 270, the simulation step 280, and the correction step 290 are executed recursively until the termination condition is met (260). When the termination condition is met, the probabilistic solver 160 outputs the value of the control input. The value of the control input defines the control policy. In other words, the value of the control input corresponds to the parameters of the control policy. The feedback controller 100 controls the device 110 according to the control policy. In particular, the feedback controller 100 controls the actuators that operate the device based on the value of the control input.

[0045] In one embodiment, the cost function 295 may include the deviation of the device 110's state from a state criterion, the deviation of the control input from an input criterion, a penalty for overtime arrival at a particular target, and so on. The cost function 295 may include the state of the device 110 in the current time instance provided by the current measurement. Additionally or alternatively, the cost function 295 may include the predicted future state of the device 110. For example, the cost function 295 may use a squared-2 norm,

number

[0046] The predicted future state x(t) of device 110 can be obtained using the simulation model 170 and the control policy u(t).

number

[0047] The optimization-based procedure shown in Figure 2 defines the constraints that define the simulation model 170 and a certain time horizon T. hor This can be implemented to minimize the cost function (1) which is based on the state of device 110 over time.

number

[0048]

number

[0049] In this context, the parameter θ of the control policy can define the control policy u(t), and the control policy can define the predicted future state x(t) of the device 110 by the simulation model 170.

[0050] The stochastic solver 160 can update the parameters of the control policy at every iteration i. The prediction model at prediction step 270 can be given by the gradient of the cost function (1) with respect to the parameters θ of the control policy.

number

number

[0051] Alternatively, the prediction model for prediction step 270 may be given by an identity model.

number

[0052] The identity model has advantages because it does not require the derivation of an analysis model and does not require the analysis model to be stored in memory 150.

[0053] Figures 4A and 4B illustrate the procedure of how the distribution of control inputs is used to determine the simulation trials for the simulation step 280 in some embodiments of the present disclosure. The distribution 410 may correspond to PDF220. The distribution 410 can define a specific number of simulation trials. For example, the distribution 410 can provide three control policies. The three control policies may be given by the first control input 420, the second control input 430, and the third control input 440. The probabilistic solver 160 uses the three control inputs 420, 430, and 440 to determine the simulation step 280. 280 This can be executed to evaluate the performance of the simulation. For example, distribution 410 has mean θ and variance σ. 2 It may also be a Gaussian distribution having [a specific characteristic].

number

[0054] Next, in correction step 290, the corrected PDF250 can be obtained using the results of simulation trials resulting from the three control inputs. For example, the control inputs / parameters that define the control policy may be updated using a weighted average of the three control inputs as follows:

number

[0055] Weights w1450, w2460, and w3470 are assigned to control inputs θ1420, θ2430, and θ3440, respectively. Weights w1450, w2460, and w3470 may be selected according to cost function 295.

number

[0056] Some embodiments are based on the understanding that different variances of different Gaussian distributions can result in different control inputs to be evaluated in simulation step 280. For example, a Gaussian distribution with high variance may result in simulation trials with control inputs that define a control policy, which are further away from the control inputs of a Gaussian distribution with lower variance.

[0057] Figure 5 shows Gaussian distributions 510, 520, and 530 with different variances according to several embodiments of the present disclosure. Gaussian distributions 510, 520, and 530 can be predicted by prediction step 270. Each of the Gaussian distributions 510, 520, and 530 has a different variance with respect to one another, but the mean 540 of the Gaussian distributions 510, 520, and 530 can be the same. Among the Gaussian distributions, the one with the highest probability, having a small variance and mean 540, may be more reliable with respect to the correct control input that defines the control policy. The different variances of the Gaussian distributions 510, 520, and 530 can result in different control inputs that define the control policy, which are evaluated in simulation step 280. For example, a Gaussian distribution 530 with a high variance may result in simulation trials with control inputs that define the control policy that are further away than the control inputs of a Gaussian distribution 510 with a lower variance.

[0058] Figure 6 is a schematic diagram showing a correction step 290 for generating a corrected PDF 250 according to some embodiments of the present disclosure. The probabilistic solver 160 corrects the PDF 220 of the predicted values ​​of the control inputs based on the PDF 240 of the simulated values ​​connected to the control inputs to generate the PDF 250 of the control input values. For example, the three control inputs 420, 430, and 440 in Figure 4, which define three control policies and result in three predicted future states of device 110, may be evaluated using a cost function (1). The three evaluations may be used to determine how to update the control inputs that define the control policies. For example, if the evaluation of the third control input 440 results in a lower cost than the first control input 420 and a lower cost than the second control input 430, the probabilistic solver 160 may determine that the control inputs defining the control policies in the next iteration are closer to the third control input 440.

[0059] The prediction step 270, the simulation step 280, and / or the correction step 290 are performed recursively until the termination condition is met.

[0060] Figure 7 shows exemplary termination conditions according to some embodiments of the present disclosure. Figure 7 shows a first PDF 710, a second PDF 720, and a surface area 730 defining the difference between the first PDF 710 and the second PDF 720. The first PDF 710 is a predicted cost relation Number The first PDF, PDF230, may correspond to the PDF defining the predicted cost function, and the second PDF, PDF720, may correspond to the PDF240 of the simulated value. The surface area, PDF730, may be used as a metric to determine whether to stop the iteration or continue the recursive iteration. For example, the surface area, PDF730 may be too large to terminate the recursive iteration in order to finish the optimization-based procedure shown in Figure 2. After further iterations, the stochastic solver, PDF160, may generate a third PDF740 and a fourth PDF750. The third PDF740 may correspond to the PDF (e.g., PDF230) defining the predicted cost function, and the fourth PDF750 may correspond to the PD of the simulated value. F( For example, this could correspond to PDF240). In this example, the surface area 760 between the third PDF740 and the fourth PDF750 may be determined to be small enough to stop the recursive iteration in order to terminate the optimization-based procedure. In some alternative embodiments, the termination condition is based on a similarity metric between the PDF of the control input value in the current iteration and the PDF of the control input value in the previous iteration. If such similarity is below a threshold, the termination condition is intended to be met.

[0061] For example, an optimization-based procedure may use the Kullback-Leibler divergence metric as the termination condition. Two distributions p i+1 (θ) and p i For (θ), the Kullback-Leibler divergence can be defined as follows:

number

[0062] Alternatively, an optimization-based procedure may use the following metric for the proximity of the control input:

number

[0063] Some embodiments use a Kalman filter as a stochastic solver. A Kalman filter is a process (or method) that generates estimates of unknown variables using a set of measurements observed over a period of time, which include statistical noise and other inaccuracies. In practice, these generated estimates of unknown variables (such as control inputs) may be more accurate than estimates of unknown variables generated using a single measurement. A Kalman filter generates estimates of unknown variables by estimating a joint probability distribution over the unknown variables. A Kalman filter is a two-step process that includes a prediction step and an update step. In the prediction step, the Kalman filter uses a prediction model to predict the current variables, along with their uncertainties, which are dominated by process noise. For example, the prediction model may be designed so that it receives process noise to reduce the uncertainty in the variables while predicting the current variables. In practice, the predicted current variables may be represented by a joint probability distribution over the current variables.

[0064] Some embodiments are based on the recognition that, because the cost function 295 and the control inputs are interdependent, the Kalman filter should collectively adjust the parameters that define the control policy. One advantage of using a Kalman filter is that the interdependence of the control inputs is accounted for by the coupling distribution of the control inputs.

[0065] Figure 8 shows implementations of the stochastic solver 160 as an iterative procedure using a Kalman filter 800 according to several embodiments of the present disclosure. According to one embodiment, the state of the Kalman filter 800 is defined by control inputs. For this purpose, the objective of the Kalman filter 800 is to iteratively generate control inputs for different control steps. In an exemplary embodiment, the Kalman filter 800 may iteratively generate control inputs using a prediction model 810 and a measurement model 840.

[0066] The prediction model 810 of the Kalman filter 800 can be used for the prediction step 270. The measurement model 840 of the Kalman filter 800 can be used as a combination of the simulation step 280 and the correction step 290. For example, the measurement model 840 of the Kalman filter 800 can be specified by the cost function 295. The cost function 295 can be interpreted as having a prior distribution given by a multivariate Gaussian distribution,

Number

[0067] Based on the observation that maximizing the logarithm of the multivariate Gaussian distribution is equivalent to minimizing the quadratic 2-norm that defines the cost function 295, the probabilistic solver 160 is based. Thus, the multivariate Gaussian distribution can define the prior distribution 850 of the cost function 295. For example, the prior distribution can be the mean h ref and the covariance Q -1 which can be given by.

Number

[0068] To generate the control input in the current iteration, the prediction model 810 can be configured to predict the value of the control input using the prior knowledge 820 of the control input. For example, the prior knowledge 820 of the control input can be a measure of how quickly the control input is expected to change between iterations or whether it is desirable to change. For example, the control input can be expected to change according to an identity model or the gradient with respect to the cost function 295 and the process noise. The process noise can be a measure of how reliable the prediction model 810 is. The prior knowledge 820 of the control input can be a joint probability distribution (or Gaussian distribution) for the control input. The process noise can be a joint Gaussian distribution with zero mean and prior covariance P0. The process noise may be artificially designed.

[0069] The predictive model 810 may generate a predicted coupling probability distribution 830 using prior knowledge of the control input 820 and the coupling probability distribution for the control input in the previous iteration 860. For example, the coupling probability distribution for the control input in the previous iteration 860 is the mean θ calculated in the previous iteration (e.g., in iteration i-1). i-1|i-1 and variance (or covariance) P i-1|i-1 It can be defined by the following. For example, the bond probability distribution for the control input in the previous iteration 860 can be generated based on the bond probability distribution generated in a previous iteration (e.g., in iteration i-2). The predicted bond probability distribution 830 for the control input is the predicted mean θ i|i-1 And the predicted variance (or covariance) P i|i-1 It can be defined by and .

[0070] The predicted coupling probability distribution 830 for the control input may be PDF220 of prediction step 270 shown in Figure 2. For example, the predicted mean θ i|i-1 and prediction So portion Dispersion (or covariance) P i|i-1 This uses the predictive model 810 together with the identity model. It can be calculated using

number

[0071] Alternatively, the predicted mean θ i|i-1 and the predicted variance (or covariance) P i|i-1 This can be calculated using the prediction model 810 along with the gradient of the cost function 295.

number

[0072] Please understand that this disclosure is not limited to the two examples of predictive models, and other predictive models may be used in a similar manner.

[0073] Different embodiments use different implementations of the stochastic solver 160. Additionally or alternatively, different embodiments use different types of Kalman filters. For example, one embodiment uses an unscented Kalman filter framework. This embodiment is advantageous because the unscented Kalman filter performs updates without estimating gradients during its correction step. Thus, this embodiment can solve optimization problems without estimating gradients, which is beneficial for some dynamic devices with complex dynamics.

[0074] Furthermore, the unscented Kalman filter can provide a sample of control inputs favorable to simulation step 280. For example, the unscented Kalman filter represents the PDF of the control input values ​​as sigma points and uses these sigma points throughout the iterations of the unscented Kalman filter to perform the unscented transformation. Several embodiments are based on the understanding that the operation can be simulated by using sigma points as sampled control inputs. Thus, the evaluation of the device's operation is integrated into the probabilistic framework of the unscented Kalman filter.

[0075] The measurement model 840 of the Kalman filter 800 can use the predicted joint probability distribution 830 to determine the simulation trials, as shown in Figure 4. Some embodiments use an unscented Kalman filter implementation to determine the simulation trials. The unscented Kalman filter computes sigma points representing the predicted joint probability distribution 830. Thus, the sigma points define an alternative definition of the predicted joint probability distribution 830. Each sigma point characterizes one simulation trial in simulation step 280.

number

[0076]

number

[0077] Given a simulated sigma-point based state trajectory, the measurement model 840 can use the cost function 295 to evaluate the simulated performance for each sigma point.

number

[0078] The weights of the unscented Kalman filter may be chosen to be different. For example, the weights may be chosen to have equal weights.

number

[0079] The sigma points can be calculated using the covariance matrix and the Koretsky decomposition.

number

[0080] Unscented Kalman filters are advantageous because the estimates of the covariance matrix and control inputs define both the simulation trials to be evaluated and how the control inputs are updated using the results of the simulation trials.

[0081] The measurement model 840 can perform the correction step 290 using the simulated performance for each sigma point.

number

[0082] Some embodiments iteratively update the control input using the sigma point and the simulated performance of the sigma point.

number

[0083] It should be understood that the presented implementation represents an illustrative implementation. In practice, other implementations such as different weight selections, different sigma point calculations, or different prediction models may also be chosen.

[0084] Next, the correction step 290 of the measurement model 840 uses the updated mean θ of the control input 870. i|i and the updated covariance P i|i This can be output. The updated coupling distribution 880 of the control inputs can then be used to determine the input signal 180 to the device 110 if the termination condition shown in Figure 7 is met. If the termination condition is not met, the next iteration 890 is performed.

[0085] Some embodiments are based on the recognition that a further advantage of having a parameterized Kalman filter for the control input is that the Kalman filter can test several samples from the PDF220 of the predicted value to accelerate convergence. Testing multiple samples results in multiple simulations. However, the tests and simulations can be performed simultaneously using multiple parallel processors. In addition, different samples can be tested using different Kalman filters with different measurement noises.

[0086] Figure 9 shows an exemplary list of the cost function 295 for the purposes of some embodiments of the present disclosure. The cost function can be any combination of the elements in the list. For example, the cost function is the cost 910 for any number of state deviations from the target state, h ref =x ref and h(x(t),u(t))=x(t); cost for any number of input deviations from the target input, h ref =u ref It can also include h(x(t),u(t))=u(t).

[0087]

number

[0088] Additionally or alternatively, it is possible to select objectives that include a cost for the specific control input itself (rather than the state or input), for example,

number

[0089] In one embodiment, the feedback controller 100 controls the robot 1010 from an initial position 1040 to a target position 1050 such that deviation from the path 1060 of the robot 1010 is minimized. For example, the feedback controller 100 collects measurements indicating the state of the robot 1010, such as the robot 1010's current position. Furthermore, the feedback controller 100 recursively runs a probabilistic solver 160 until a termination condition is met to generate control inputs that maintain the robot 1010's position along the path 1060. The feedback controller 100 further controls the robot 1010 based on the generated control inputs.

[0090] In addition, in some embodiments, the feedback controller 100 may be used to control the crane operating the load. For example, the feedback controller 100 may minimize vibrations of the crane load relative to a reference path while operating the load.

[0091] This description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of exemplary embodiments provides a practicable description for realizing one or more exemplary embodiments. The intent is to describe various modifications that may be made in the function and configuration of the elements without departing from the spirit and scope of the subject matter disclosed as described in the claims.

[0092] The following description provides specific details for a complete understanding of the embodiments. However, it will be understood by those skilled in the art that embodiments may be carried out without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other examples, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments. Furthermore, similar reference numbers and names in different drawings indicate similar elements.

[0093] Furthermore, individual embodiments may be described as processes shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are complete, but it may have additional steps that are not discussed or included in the diagrams. Moreover, not all operations in any particular process described may occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the function may correspond to the function's return to the calling function or the main function.

[0094] Furthermore, embodiments of the disclosed subject matter may be implemented at least partially manually or automatically. Manual or automatic implementations may be performed using, or at least assisted by, a machine, hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. If implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the required tasks may be stored in a machine-readable medium. The required tasks may be performed by a processor.

[0095] The various methods or processes outlined herein may be coded as software executable on one or more processors using any one of various operating systems or platforms. In addition, such software may be written using any of several preferred programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of program modules may be combined or distributed as desired in various embodiments.

[0096] Embodiments of this disclosure may be embodied as an example provided. The actions performed as part of the method may be ordered in any preferred manner. Thus, embodiments may be constructed in which the actions are performed in a different order than exemplary, which may include performing several actions simultaneously that are shown as sequential actions in the exemplary embodiments.

[0097] Furthermore, embodiments of the present disclosure and the functional operations described herein may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including structures disclosed herein and their structural equivalents, or one or more combinations thereof. Furthermore, some embodiments of the present disclosure may be implemented as one or more modules of computer program instructions encoded on a tangible, non-temporary program carrier for execution by a data processing device or for controlling the operation of a data processing device. Furthermore, program instructions may be encoded on artificially generated propagating signals, such as mechanically generated electrical, optical, or electromagnetic signals generated to encode information for transmission to a suitable receiving device for execution by a data processing device. Computer storage media may be machine-readable storage devices, machine-readable storage boards, random or serial access memory devices, or one or more combinations thereof.

[0098] According to embodiments of this disclosure, the term “data processing device” can encompass all types of devices, machines, and apparatus for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may include dedicated logic circuits, such as FPGAs (field-programmable gate arrays) or ASICs (application-specific integrated circuits). In addition to hardware, the device may also include code that generates the execution environment for the computer program, such as processor firmware, a protocol stack, a database management system, an operating system, or code comprising one or more of these.

[0099] Computer programs (also called, or may be described as, programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed as standalone programs or in any form, including modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs may, but are not required, correspond to files in a file system. A program may be stored in part of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple collaborative files, for example, a file that stores one or more modules, subprograms, or parts of code.

[0100] Computer programs can be deployed to run on a single computer, located in a single site, or distributed across multiple sites and interconnected by a communication network. A computer suitable for running a computer program may include, for example, a general-purpose microprocessor, a dedicated microprocessor, or both, and any other type of central processing unit, or be based on them. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a central processing unit for executing instructions, and one or more memory devices for storing instructions and data.

[0101] Generally, a computer may also be coupled in a manner that allows it to include or operate on one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or to receive data from them, transfer data to them, or both. However, a computer does not need to have such devices. Furthermore, a computer can be incorporated into other devices, to name a few, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices, such as Universal Serial Bus (USB) flash drives.

[0102] To provide user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, thereby allowing the user to provide input to the computer. Other types of devices may also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic input, speech input, or tactile input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from devices used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0103] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component such as an application server, or a frontend component, such as a client computer having a graphical user interface or a web browser on which a user can interact with the implementation of the subject matter described herein, or any combination of one or more such backend components, middleware components, or frontend components. The components of the system may be interconnected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0104] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact over a communication network. The relationship between a client and a server arises from computer programs running on each computer that have a client-server relationship with each other.

[0105] While this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications can be made within the spirit and scope of this disclosure. Therefore, it is the nature of the claims to encompass all such variations and modifications that fall within the true spirit and scope of this disclosure.

Claims

1. A feedback controller for controlling the operation of a device in different control steps based on feedback signals including measured values ​​indicating the state of the device in the different control steps, comprising at least one processor and a memory storing instructions, wherein when an instruction is executed by the at least one processor, the feedback controller receives instructions for the control step. In the control step, the device is made to collect measured values ​​indicating the state of the device. The parameterized stochastic solver is run recursively on the control inputs to the actuators that operate the device until the termination condition is met, generating control inputs for the control steps, and during each execution, the stochastic solver, The system is configured to estimate the PDF of the predicted value of the control input from the probability distribution function (PDF) of the control input value using a prediction model. The system is configured to evaluate the cost function of the optimal control problem for controlling the device and generate a performance metric for the device's operation based on a simulation of the device's operation, using the measured values ​​and the values ​​of the control input sampled from a PDF of the predicted values ​​of the control input. The device is configured to estimate the PDF of the simulated value connected to the control input based on a measurement model that connects the performance metric of the operation of the device to the control input, Based on the PDF of the simulated value connected to the control input, the PDF of the predicted value of the control input is corrected to generate the PDF of the value of the control input, and the instruction, when executed by the at least one processor, further, to the feedback controller for the control step, A feedback controller that controls the actuator that operates the device using the average of the PDFs of at least the values ​​of the control inputs.

2. The feedback controller according to claim 1, wherein the prediction model is an identity model.

3. The feedback controller according to claim 1, wherein the prediction model is given by the gradient of the cost function with respect to the control input.

4. The feedback controller according to claim 1, wherein for each iteration of the iterations performed for each of the control steps, the probabilistic solver is further configured to evaluate the cost function for a plurality of samples that define simulation trials of the PDF of the predicted value.

5. The feedback controller according to claim 4, wherein, in order to generate the PDF of the simulated value connected to the control input from a plurality of performance metrics, the probabilistic solver is further configured to determine a weighted combination of evaluations of a plurality of samples of the PDF of the predicted value.

6. The feedback controller according to claim 4, wherein the cost function for multiple samples of the PDF of the predicted values ​​is evaluated in parallel using multiple processors.

7. The feedback controller according to claim 1, wherein the termination condition is based on a similarity metric between the PDF of the value of the control input and the PDF of the value of the control input in the previous iteration.

8. The feedback controller according to claim 1, wherein the stochastic solver is a Kalman filter.

9. The feedback controller according to claim 1, wherein the probabilistic solver is an unscented Kalman filter that performs gradient-free correction of the predicted value of the control input.

10. The feedback controller according to claim 9, wherein the unscented Kalman filter is configured to generate a PDF of the simulated value connected to the control input using an evaluation of the cost function in order to update the PDF of the predicted value using the gradient-free correction.

11. The feedback controller according to claim 9, wherein the unscented Kalman filter evaluates the cost function for a plurality of sigma points of the PDF of the predicted value.

12. The feedback controller according to claim 11, wherein the unscented Kalman filter is configured to generate the sigma points based on the predicted mean of the PDF of the predicted values ​​and the covariance matrix of the PDF of the predicted values, in order to determine the sigma points.

13. The feedback controller according to claim 12, wherein the aforementioned sigma point is determined based on Cholesky decomposition.

14. A method for controlling the operation of a device in different control steps based on a feedback signal including a measured value indicating the state of the device in the different control steps, In the control step, the measurement values ​​indicating the state of the device are collected, The process includes recursively running a parameterized stochastic solver on the control inputs to the actuators that operate the device until a termination condition is met, generating control inputs for the control step, during each execution, the stochastic solver, The system is configured to estimate the PDF of the predicted value of the control input from the probability distribution function (PDF) of the control input value using a prediction model. The system is configured to evaluate the cost function of the optimal control problem for controlling the device and generate a performance metric for the device's operation based on a simulation of the device's operation, using the measured values ​​and the values ​​of the control input sampled from a PDF of the predicted values ​​of the control input. The device is configured to estimate the PDF of the simulated value connected to the control input based on a measurement model that connects the performance metric of the operation of the device to the control input, The method is configured to correct the PDF of the predicted value of the control input based on the PDF of the simulated value connected to the control input, and to generate the PDF of the value of the control input, and the method further, A method comprising controlling the actuator that operates the device using the average of the PDFs of at least the values ​​of the control inputs.

15. The method according to claim 14, wherein the prediction model is an identity model.

16. The method according to claim 14, wherein the prediction model is given by the gradient of the cost function with respect to the control input.

17. The method according to claim 14, wherein for each iteration of the iterations performed for each of the control steps, the probabilistic solver is further configured to evaluate the cost function for a plurality of samples that define simulation trials of the PDF of the predicted value.

18. The method according to claim 17, wherein the probabilistic solver is further configured to determine a weighted combination of evaluations of multiple samples of the PDF of the predicted value in order to generate the PDF of the simulated value connected to the control input from a plurality of performance metrics.

19. The method according to claim 14, wherein the termination condition is based on a similarity metric between the PDF of the value of the control input and the PDF of the value of the control input in the previous iteration.

20. A non-temporary computer-readable storage medium embodying a processor-executable program for performing a method for controlling the operation of a device in different control steps based on feedback signals including measured values ​​indicating the state of the device in the different control steps, wherein the method is In the control step, the measurement values ​​indicating the state of the device are collected, The process includes recursively running a parameterized stochastic solver on the control inputs to the actuators that operate the device until a termination condition is met, generating control inputs for the control step, during each execution, the stochastic solver, The system is configured to estimate the PDF of the predicted value of the control input from the probability distribution function (PDF) of the control input value using a prediction model. The system is configured to evaluate the cost function of the optimal control problem for controlling the device and generate a performance metric for the device's operation based on a simulation of the device's operation, using the measured values ​​and the values ​​of the control input sampled from a PDF of the predicted values ​​of the control input. The device is configured to estimate the PDF of the simulated value connected to the control input based on a measurement model that connects the performance metric of the operation of the device to the control input, The method is configured to correct the PDF of the predicted value of the control input based on the PDF of the simulated value connected to the control input, and to generate the PDF of the value of the control input, and the method further, A non-temporary computer-readable storage medium that includes controlling the actuator that operates the device using the average of the PDF values ​​of at least the control inputs.

Images