A framework for optimal hardware-software co-design of control-on-a-chip systems

Abstract

A system has a device capable of performing a physical operation, the physical operation having quantifiable measures of its controlled performance, a controller communicatively coupled to a computing device to receive selected controller parameters and generate control inputs for the device, and the computer device having one or more processors configured to execute code to cause the one or more processors to: iteratively receive sets of measurements of the quantifiable performance measures when the device is operating under control of the controller; operate a deep neural network (DNN) for each set of the measurements until performance of the controller is determined to be acceptable, the DNN configured to determine a control policy for the controller; and embed the control policy by programming the controller when the performance is acceptable. The system may involve a cold atmospheric pressure plasma system used for processing materials.

Description

Patent Application U Cal No. BK-2024-106-2-PCT MN No.407869-0192 A FRAMEWORK FOR OPTIMAL HARDWARE-SOFTWARE CO-DESIGN OF CONTROL-ON-A-CHIP SYSTEMS CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to and the benefit of US Provisional Application No. 63 / 574,010 filed April 3, 2024, which is incorporated herein by reference in its entirety. STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT

[0002] This invention was made with Government support under Grant Number 2317629 awarded by the National Science Foundation. The Government has certain rights in this invention. TECHNICAL FIELD

[0003] This disclosure relates to hardware and software co-design, more particularly to a framework for automated design and tuning of arbitrary control policies on arbitrary hardware. BACKGROUND

[0004] The digital age has made embedded control a key component to user-oriented, portable, and internet-of-things devices. In addition, with emergent complex systems arises the need for advanced optimization-based control strategies like model predictive control. However, the unified implementation of these advanced strategies on hardware remains a challenge. Designing complex control policies for embedded systems is inherently an interwoven process between the algorithmic design and hardware implementation, which will require a hardware-software co-design perspective.

[0005] Embedded systems lie at the core of many online control systems technologies, including autonomous systems, IoT systems, and biomedical devices, amongst others.Microcontrollers / microprocessors (MCUs) have played a major role in enabling embedded control for portable devices. Due to their widespread availability and adoption, MCUs have been designed such that they can be easily programmed using high-level programming languages, such as C and Python, and be deployed with relative ease. Meanwhile, emerging technologies have increasingly complex dynamics and typically rely on advanced model- based control strategies, such as model predictive control (MPC), that can handle multivariable dynamics and state and input constraints. Generally, MPC comprises a control strategy that uses a model of the system to predict the future behavior of the system over a finite time horizon to minimize a cost function, while satisfying constraints.

[0006] One should note that MCUs here may be referred to as controllers to avoid any limitation to any particular kind of controller. The controller may comprise an Application Specific Integrated Circuit (ASIC), FPGA, microcontroller, microprocessor, or other control chips.

[0007] As a result, significant efforts have targeted the development of automated software- based code generation tools for fast numerical optimization on MCUs. These tools, such as ACADO (automatic control and dynamic optimization), GRAMPC (gradient based augmented Lagrangian approach), and FORCES (software platform for real-time numerical optimization used to solve MPC problems) utilize tailored implementations of structured formulations of the underlying optimization problem to efficiently compute the optimization solution for real-time control. Examples include, but are not limited to using sequential quadratic programming, nonlinear interior point methods, or accelerated gradient methods. The structured formulation that these code generation tools require can make the implementation of robust and learning-based optimal control strategies more challenging, while the focus on solely the software side implies that the form of hardware implementation is also limited. Furthermore, the design of embedded controllers relies on more than just thefast solution of the optimal control problem. It also involves maintaining numerical robustness at low computational accuracy, tolerance against infeasibility, low code complexity, and low memory / resource utilization, among other considerations.

[0008] Many of these challenges require explicit knowledge of the hardware specifications, as well as knowledge of how computations are performed and accelerated on the hardware. As advanced hardware technologies, such as graphics processing units (GPUs), field programmable gate arrays (FPGAs), tensor processing units (TPUs) become commonplace, the principle of hardware-software co-design will become an integral consideration for the physical implementation of embedded controllers. Hardware-software co-design involves the concurrent design of the control algorithm (software) and its embedded implementation (hardware).

[0009] Optimized control policy tuning is often a tedious and cumbersome process, which is further exacerbated by the extensive workflow to go from the programmatic control policy design to embedded implementation. Control policy auto-tuning (aka calibration) is well- established for simple controllers, but auto-tuning for generic control structures has recently regained traction. One popular approach to auto-tuning uses principles of reinforcement learning (RL). To this end, policy-gradient RL methods involve updating the control policy parameters via gradient descent. While policy gradient is a scalable approach, it can require many evaluations on the true system and can be prone to getting stuck at local optimizers.

[0010] Another popular approach to auto-tuning relies on data-driven optimization, particularly Bayesian optimization (BO). Autotuning can be interpreted as a black-box problem where the objective function is expensive to evaluate, potentially non-convex, and without closed-form derivatives. BO is a “global” optimization method that takes a probabilistically principled approach to reduce the number of interactions with the real system. Variants of BO for auto-tuning are also emerging; however, most of these studies, donot consider the hardware considerations for embedded control. Even still, the focus of approaches that do consider hardware remains on the hardware-constrained optimization (i.e., real-time computations) of the software, rather than hardware-software codesign. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG.1 shows a flowchart of a current control-on-a-chip design process.

[0012] FIG.2 shows a flowchart of an embodiment of a control-on-a-chip design process.

[0013] FIG.3 shows a diagram of an embodiment of a data-driven optimization framework.

[0014] FIG.4 shows an image and schematic for an atmospheric pressure plasma jet (APPJ).

[0015] FIG.5 shows a graphical representation of a confusion matrix for a hardware feasibility classifier.

[0016] FIGs.6A-6B show graphs of data from observed closed-loop performance of plasma treatments overall and in a zoomed-in version.

[0017] FIG.7 shows a graph of hypervolume improvements for five replicates of multi- objective Bayesian optimization (MOBO) and random search for control-on-a-chip designs for an APPJ.

[0018] FIGs.8A-8C show closed-loop trajectories of one replicate of MOBO for three different numbers of iterations.

[0019] FIG.9 shows a graph of observed performance metrics for three replicates of MOBO for control-on-a-chip design. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The embodiments herein describe an end-to-end framework for the automated design and tuning of arbitrary control policies on arbitrary hardware. The framework relies on deep learning as a universal control policy representation and multi-objective Bayesian optimization (BO) to facilitate iterative systematic controller design. The large representation power of deep learning and its ability to decouple hardware and software design are a centralcomponent to determining feasible control-on-a-chip policies. Then, BO provides a flexible sequential decision-making framework where practical considerations such as multi-objective optimization concepts and categorical decisions can be incorporated to efficiently design embedded control policies that are directly implemented on hardware. The embodiments demonstrate the proposed framework via closed-loop simulations and real-time experiments on an atmospheric pressure plasma jet for plasma processing of biomaterials.

[0021] The hardware-software co-design paradigm presents a new take on the embedded control design problem. Formally, the discussion here denotes this new perspective of hardware-software co-design as control-on-a-chip (CoC) design since the embodiments aim to provide a unified workflow to place arbitrary control policies on arbitrary hardware. The term “arbitrary” as used here means that the approach does not rely on specifics of the hardware when testing the control policy, making them both arbitrary and therefore usable with many different types of hardware. The embodiments here pose the CoC co-design problem as an optimization problem that incorporates a hardware feasibility constraint and multiple levels of decisions to create an all-in-one framework.

[0022] First, the embodiments establish a flexible workflow that takes advantage of recent advances in deep learning. Deep neural networks (DNNs) have played a pivotal role in imitation learning of MPC (model predictive control) policies and differentiable predictive control. The embodiments use the imitation learning perspective as it poses two key advantages: (i) it provides a “physically interpretable” control policy; and (ii) by virtue of (i), it reduces the design parameter space. The proposed CoC workflow of the embodiments consists in (i) designing an expert control policy (based on optimization-based control strategies), (ii) using DNNs to represent the expert control policy, and (iii) implementing the DNN-based control policy on hardware.

[0023] The embodiments then use multi-objective Bayesian optimization (MOBO), discussed in D. Hernandez-Lobato, J. Hernandez-Lobato, A. Shah, and R. Adams, “Predictive entropy search for multi-objective Bayesian optimization,” in Int. Conf. Mach. Learn., pp.1492– 1501, PMLR, 2016, to encapsulate the multi-step CoC design workflow to create an end-to- end optimization framework. Solving a multi-objective CoC design problem via MOBO entails finding an optimal set of control policies rather than one single optimizer, which allows a practitioner to choose the best design(s) according to the needs or preferences of the application. Furthermore, BO offers flexibility when incorporating design choices from each step of the CoC workflow.

[0024] To begin, one must first seek optimal CoC policies that minimize a set of closed-loop cost metrics subject to hardware constraints. In general, CoC policies can be represented as a space of all possible machine code instructions that can be executed on a given choice of hardware. Since this is a complex design space involving decisions made by human experts, the design problem is often decomposed into two key steps: (i) choice of a high-level control program θ that must reside in the space of possible programs Θ (generally very high- dimensional and complex); and (ii) choice of a code generation strategy that translates θ into executable machine code, which has its own set of design parameters denoted by γ ∈ Γ (e.g., numerical representation and parallelization options). One can formulate the search for an optimal pair of program and code generation parameters (θ⋆, γ⋆) in terms of the following multi-objective optimization (MOO) problem:g(θ, γ) = 1, (1f) ^^^^(θ, γ) ∈ Θ, Γ, (1g)where (1a) represents the set of M closed-loop performance metrics; (1b) represents the system dynamics that describes the evolution of the system state xtin response to control actions utand disturbances wtat time t; (1c) defines the CoC policy π(xt; θ, γ) that maps (measured or estimated) states to control actions for a specific choice of program and code generation parameters; (1d) is a stochastic disturbance that evolves according to some probability distribution ^^^^^^^^^^^^that is conditionally independent of previous disturbance realizations given the current states and actions; (1e) represents the i-th performance metric Jidefined in terms of the closed-loop system evolution given local stage costand terminal cost ℓT,ifunctions over a finite time horizon T; (1f) denotes a hardware resource utilization constraint represented by a binary function g : Θ × Γ → {0, 1} that indicates if the high-level control program can be compiled and executed on the available hardware (1) or not (0); and (1g) represents the user-defined search space of control programs and code generation strategies. The expectation ^^^^{ ∙} in (1e) is taken with respect to the stochastic disturbance sequence of the closed-loop system {w0, ... ,wT−1}. The discussion may refer to this as a stochastic variant of the MPC.

[0025] The MOO problem (1) represents a very general framework for CoC design. In fact, one can interpret virtually all end-to-end control design procedures as an instance of (1). However, approximations are necessary in practice due to the intractability of (1), which stems from two main challenges. First, the program parameter space Θ is abstract due to the choice of an appropriate control policy representation. To ensure it has sufficiently large representation power, the control policy will typically be embedded in some high- dimensional space Θ ⊂ ℝD, where D can be very large. Furthermore, this space may involvediscrete and continuous variables that are needed to represent logical relationships between different sets of variables. Second, one often does not have exact knowledge of the system dynamics f(·), disturbance distribution ^^^^^^^^^^^^, and the hardware utilization constraint function g(·), which prevents the application of traditional MOO methods that require equation- oriented forms for all objective and constraint functions.

[0026] Standard control design approaches often proceed in the following steps that are illustrated in FIG 1. The process begins at 10 with a high-level control program selected at 12. This may involve restricting the control policy representation by constraining Θ to describe a narrow set of policies in a low-dimensional space and evaluating its performance. The process then independently searches for control program parameters θ that approximately minimize the closed-loop performance metrics either using approximate models, or (limited) closed-loop data. The process then selects a code generation strategy and evaluates it based on its ability to be successfully implemented while matching the performance of the high- level program, this may involve a search for code generation parameters γ that enable the desired θ to be executed on the available hardware. If it fails at 18, then one must go back and repeat the process again for a narrower set of more computationally tractable policies.

[0027] For example, complex control policy formulations as in optimization-based control may require specific code libraries and / or routines that are not easily implemented on low- resource hardware due to memory restrictions, or reduced accuracy due to quantized numeric representation. Furthermore, emergence of advanced hardware (e.g., GPUs, FPGAs) to speed up computations requires additional low-level code generation that requires specialized knowledge of the device architecture. Separate design of θ and γ misses important interactions between high-level program representation and code generation. In particular, the hardware utilization constraint function g(·) represents whether or not the policy can be embedded into the hardware or can satisfy timing constraints.

[0028] Examples of parameters that influence this constraint include numeric representation, number of operations, parallelization options, etc. Without co-design, there is a much greater chance that CoC policies cannot be implemented as knowledge of how the algorithmic requirements are translated to the physical wires in hardware are not generally accessible.

[0029] The embodiments here develop an iterative learning-based strategy capable of systematically tackling the hardware-software co-design problem (1). The core structure of the proposed strategy, which relies on deep learning to simplify the policy and hardware utilization constraint, is presented below. The discussion below describes an efficient multi- objective black-box optimization strategy that takes advantage of this structure by searching over a reduced set of parameters.

[0030] The lack of an easy-to-search program space Θ and known structure for the feasibility constraint g greatly complicates the traditional CoC design procedure. Both of these problems can be addressed by working with deep neural network (DNN) policies such that θ = {θW, θA} can be separated into continuous weight and bias parameters θWand architecture parameters θAthat can be discrete. This not only helps simplify the learning process for g, but also allows the process to take advantage of prior knowledge to train θWsuch that it only considers a small subset of parameters when optimizing closed-loop performance.

[0031] Deep learning is a generalized term for computational structures / graphs characterized by multiple “layers.” Through multiple layers, deep learning transforms an input representation to abstract representations until the ultimate output is learned. It is exactly the many layers that allow deep learning to extract (on its own) features from raw data that are relevant to learning control policies.

[0032] The key advantage of DNN policies is the surprisingly robust ability to train such large structures using (stochastic) gradient descent style methods. For simplicity ofpresentation, consider a fully connected feedforward DNN control policy πdnn(x; θ) with L hidden layers and H nodes per layer, which can be mathematically defined as follows: πdnn(x; θ) = αL+1◦ βL◦ αL◦・・・ β1◦ α1(x), (2) where α1(x) = W1x + b1is an affine transformation of the input, αl(zl−1) = Wlzl−1+ blare affine transformations of the hidden layers for all l ∈ {2, ... ,L + 1}, βl(z) are nonlinearactivation functions (e.g., ^^^^^^^^(^^^^) = max{^^^^, 0} for ReLU activation functions) for all l ∈ {1, . . .,L}, and θW= {W1, b1, ... ,WL+1, bL+1} denotes the collection of weights and biases that parameterize the network for a fixed architecture θA(e.g., type of activation function, L, and

[0033] Due to their continuous representation, θWcan be trained by minimizing a loss function that captures how well the DNN performs on a given task. This process is known to work well in practice under the assumption that the gradient of the loss function with respect to θWcan be efficiently computed via backpropagation. This is not necessarily the case when one attempts to use πdnnin (1) unless a differentiable structure for the dynamics and cost functions is known.

[0034] Note that the DNN policy defined in (2) is just one choice of architectural representation of artificial neural networks. In fact, any deep learning architecture can be used to approximate (2). For example, if the state or any other exogenous signals involved image data, one could exploit a convolutional neural network structure that is designed to specifically exploit the regularity of image pixel patterns.

[0035] Given that the ideal policy is represented by a DNN, in addition to the program (i.e., software) (2), CoC policies also require specification of the embedded version of the program that can be run on the actual hardware. Although this difference is not practically important in the absence of resource limitations, it is very important in cases where there are constraintson the number of real-time computations and / or resource (memory or power) utilization. A useful property of DNNs is that the resource utilization is the same for all θWgiven a fixed architecture θA. To see this, one can compute the number of operations Nopfor a dense DNN of the form (2) as Nop= (nin+ 1)H + H(H + 1)L + (H + 1)nout, (3) where ninand noutare the number of inputs and outputs, respectively. Thus, by simply changing the architecture of the DNN (e.g., reduce number of nodes or layers), the process can lower the evaluation cost on hardware.

[0036] Nonetheless, one cannot use Nopto directly characterize the feasible set of CoC policies, i.e., Ƒ = {(θ, γ) ∈ Θ × Γ : g(θ, γ) = 1}, since this set will depend on how the operations written in a mid- or high-level programming language (e.g., Matlab, Python, C) get translated to low-level machine code (e.g., assembly language, binary), compiled, and then packaged. As used here, the term “high-level” programming language applies to any language that is not machine code, such as assembly language, binary, etc. Automatic code generation tools aim to provide a streamlined means of performing such tasks by abstracting the laborious translation process. As such, code generation serves as a bridge between human interpretable code and machine-interpretable instructions. In this work, the process treats code generation as a black box function π(・; θ, γ) = ^^^^^^^^(πdnn(・; θ), γ) (4) that takes as input a DNN policy and some parameters related to the translation process γ and returns a machine interpretable policy. Since changing θWwill not fundamentally change the structure of the returned CoC policy π, function g will be independent of θW and, thus, one only needs to learn an approximation of g(θA, γ). One way to learn an approximation ^^�^^ ≈ g is to run the code generation process ^^^^^^^^ for a randomly generated DNN for several values of(θA, γ) ∈ ΘA× Γ, which is expected to be a much lower-dimensional space than ΘW. The outcome can be recorded as either a successful compilation 1, or failed compilation 0. This labeled data can then be used to train a binary classifier that is capable of predicting if a new choice of architecture and code generation parameters is feasible or not. To this end, any classifier type can be used, for example, support vector machines and DNN with a sigmoid activation function at the output layer. The discussion highlights that the major advantage of this approach is that ^^�^^ can be trained independently of the quality of the CoC policy. Not only can this process be done fully offline, but also it does not require any system data to be generated – all that is required is access to the hardware and code generation process.

[0037] The above discussion presented an efficient way to verify if a CoC policy will be feasible. Yet, the original MOO problem (1) can still be computationally intractable since θWremains a high-dimensional space with possibly thousands or more independent parameters. The question addressed here is how the search over this space can be efficiently performed without sacrificing the achieved closed-loop performance. To this end, the process relies on a class of control policies that are implicitly defined in terms of a set of interpretable equations. Specifically, the approach looks to use policies defined by an optimization problem: πopt(x;λ) = ^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^V(x, u; λ), (5a) s.t. hi(x, u; λ) ≤ 0, i = 1,... , k, (5b) gi(x, u, λ) = 0, i = 1, ... , r, (5c)where V : ℝ^^^^^^^^ × ℝ^^^^^^^^ × ℝ^^^^^^^^ → ℝ is the objective function, hi : ℝ^^^^^^^^ × ℝ^^^^^^^^ × ℝ^^^^^^^^ → ℝare inequality constraints for all i = 1, . . . , k, gi : ℝ^^^^^^^^ × ℝ^^^^^^^^ × ℝ^^^^^^^^ → ℝ are equalityconstraints for all i = 1, ... , r, and λ ∈ Λ ⊂ ℝ^^^^^^^^are tunable policy parameters. As discussed in J. A. Paulson, F. Sorourifar, and A. Mesbah, “A tutorial on derivative free policy learningmethods for interpretable controller representations,” in Proc. Am. Control Conf., pp.1295– 1306, 2023, this representation captures a large set of policies, including approximate dynamic programming and model predictive control. The key idea behind (5) is that (5a) describes some type of value or reward function, (5b) represents critical state and / or input constraints, and (5c) represents a (possibly physics-based) model of the system. A significant advantage of the structure (5) is that it provides a natural way for users to incorporate prior knowledge about the system, when available, by properly selecting or constraining the functions V, h1, ... , hk, g1, ... , gr. The central notion is that λ can be much lower dimensional than θW, such that one can derive an explicit value for the DNN parameters that depend on λ by minimizing the error between the DNN policy πdnnand the “physics informed” expert policy πopt.^^^^∗^^^^ (^^^^, ^^^^^^^^) = (6)where {(x(i), πopt(x(i); λ))}nsi=1represent a set of nsstate-action pairs acquired by solving the optimization problem (5) offline for specific state values and fixed λ values. This dataset can be generated in a variety of ways, including random sampling in the state space or using closed loop “rollouts” from likely initial conditions. It is important to note that special care must be taken in generating the dataset to train the DNN as the approximation of (5) may reduce the robustness properties ensured by implementing (5) directly depending on the quality of the overall training process. In accordance with the universal approximation theorem, there exists a DNN that matches (5) exactly. As such, given a sufficiently large architecture and enough training data, one can ensure πdnn(·;^^^^^∗^^^(λ, θA), θA) → πopt(·; λ) for some θA.

[0038] Notice that the solution to (6) will depend on both the expert policy parameters λ and the DNN architecture hyperparameters θA. Therefore, the proposed CoC policy has the following unique structure πCoC(∙; λ, θA, γ) = ^^^^^^^^ (πdnn(^^^^^∗^^^(λ,θA), θA), γ), (7) which depends on three sets of parameters, mainly λ, θA, and γ that all appear in different components of the CoC framework. An illustration of the proposed CoC design process is shown in FIG.2, which is based on selecting (λ, θA, γ) ∈ Λ×ΘA×Γ to optimize closed-loop performance metrics of interest. However, one does not have a closed-form expression for how performance metrics depend on (λ, θA, γ). This is further compounded by the cost of collecting closed-loop performance data since it requires (i) training a DNN policy, (ii) executing a code generation process to run the policy on embedded hardware, and (iii) running hardware-in-the-loop closed-loop experiments to collect performance data. Next, the process presents an efficient procedure for searching over this joint parameter space.

[0039] In FIG.2, the process starts at 30, and the process as shown has two subdivisions, a first software subdivision, “high-level control program selection,” and “hardware implementation.” Within the high-level control program selection, first one selects and evaluates a “physics-informed” control design at 32. As used here, the term “physics- informed” means that the control policy, design, or other element takes into account the physical principles of an environment in which the device implementing the control policy operates. The physical principles may include a mathematical description of system dynamics, constraints, and / or control objectives. As an example, without limitation, a physics-informed control policy in the example of an atmospheric pressure plasma jet, discussed later, takes into account the physical environment and limitations of the plasma jet. At 36, the process creates a deep learning-based policy in pursuit of hardware-compatibility. This policy representation undergoes evaluation at 40. This process of training and testing thedeep neural network may occur iteratively until the performance is deemed acceptable. In some embodiments, the performance of the DNN being acceptable means that the DNN’s imitation of the control policy has a low enough margin of error. The hardware implementation occurs at 42 if the performance meets the requirements at 44. The process then evaluates the performance of the hardware at 44. If, it meets the standard, the process ends at 46. This process then iterates to adjust the design until the performance meets a desired performance.

[0040] The acceptability of performance of the controller as used here will generally involve one or more operating parameters of the device under control of the controller meeting some threshold or standardized value for the parameter.

[0041] The system dynamics and CoC policy together form a stochastic process due to potential uncertainty in the initial condition x0and disturbances {wt}t≥0that are random variables. xt+1= f(t, xt; ut, wt), t = 0, 1, ... , (8a) ut= πCoC(xt; ξ), (8b)where ^^^^ = (^^^^, θ^^^^, ^^^^) is the concatenation of all software and hardware parameters that definethe policy. A specific choice of ξ can be judged according to the set of M’s expected closed- loop performance metrics, i = 1, ... , M defined in (1) with the state and input sequences generated by (8). Sinceare defined as expectations over closed-loop trajectories, they only depend on ^^^^ that is of much lower dimensional than the original θ space, as discussed above. Thus, one can now pose (1) as a more manageable MOO problem:where Ξ = {(λ, θA, γ) ∈ Λ × Θ^^^^ × Γ : ^^�^^(θA, γ) = 1}is the space of CoC parameters that canbe compiled on the available hardware. Since the functions {^^^^^^^^}^^^^^^=^^1 are black-box in nature, the process resorts to derivative-free optimization (DFO) methods to approximately solve (9)in practice. In particular, the DFO method must be able to handle noisy, expensive evaluations of {^^^^^^^^}^^^^^^=^^1. The evaluations are expensive due to the need to collect hardware-in- the-loop data, as discussed previously. These evaluations will also be subject to noise due to the expectation operator that defines Ji. In practice, the process can approximate this expectation using a random sampling technique (e.g., Monte Carlo sampling) such that yi = Ji(ξ)+ εi, where εiis the effective measurement noise in the i-th performance function. Assuming K independent random samples are used to approximate the performance functions, then it is known that εiapproaches a zero-mean Gaussian random variable whose variance decreases at a rate of 1 / K by the central limit theorem.

[0042] The discussion briefly highlights the fact that the only major assumption made in (9) is that one can generate independent noisy measurements of the closed-loop performance functions. The process does not require any specific knowledge of the dynamics, or uncertainty distribution, which makes the proposed hardware-software co-design approach broadly applicable. However, the more knowledge that one can exploit in the specification of the expert policy (5), the better the choice of the ξ ∈ Ξ space, which can simplify the process of solving (9).

[0043] Since {^^^^^^^^}^^^^^^=^^1 are noisy, expensive functions defined over a relatively low- dimensional space ξ ∈ Ξ, Bayesian optimization (BO) is a natural choice of DFO framework for (9) since it is specifically designed for such cases. Furthermore, BO has been shown to surpass state-of-the-art performance in real-world controller tuning applications with a variety of policy types. BO falls under the paradigm of active learning, meaning it translates the optimization task into an iterative learning task. There are two major components in BO. First, the process constructs a probabilistic surrogate model, typically a Gaussian process (GP), to provide a posterior distribution ℙ{^^^^|^^^^^^^^} over the unknown true vector-valuedfunction values J(ξ) = . . . , JM(ξ)) given a prior dataset ^^^^^^^^ = { Second, theprocess defines an acquisition function αn: Ξ → R that uses the surrogate model to assign a utility value to the future candidate points at which one can evaluate the true function. Thus, for a well-designed αn, one would preferentially sample at a point that produces the highest possible value. The active learning process is then defined by: ^^^^^^^^+1 = argmax^^^^^^^^(^^^^). (10) ^^^^∈Ξ Since the surrogate approximation of J is expected to be much cheaper than the true function, one can (approximately) solve (10) using established optimization algorithms.

[0044] While GPs are the standard surrogate model-of-choice for BO, they are known to scale poorly with the number of data points ^^^^, requiring O(D3) floating point operations for exact inference; higher number of data points may be required to obtain representative models for higher-dimensional problems. However, there are recent advances that reduce the computational cost at the expense of accuracy.

[0045] Since the interest is in MOO, the process may not have a single best solution and instead to provide the control practitioner with an estimate of the set of Pareto optimal solutions. A point ξ ∈ Ξ is considered to be Pareto optimal if improvement in one objective means deteriorating one or more of the others. The so-called Pareto frontier is the set of Pareto optimal points, which is mathematically defined aswhere J(ξ′)≻ J(ξ) implies the pointthe point ξ, which occurs if Ji(ξ′)for all i = 1, ... ,M. To derive αn, it would be desirable to select points that grow the understanding of ^^^^⋆. The process may use the expected hypervolume improvement (EHVI) acquisition function defined as follows: ^^^^^^^^(^^^^) = ^^^^{^^^^^^^^(^^^^ ∪ {^^^^(^^^^)}, ^^^^) − ^^^^^^^^ (^^^^, ^^^^)} (12)where HV(^^^^, r) denotes the hypervolume of a finite approximate Pareto set ^^^^ and a reference point r ∈ ℝMthat bounds ^^^^ from below. The HV can be computed exactly as the M-dimensional Lebesgue measure:where ^^^^ = {y(1), ... , y(q)} is composed of a finite set of q points. EHVI is notoriously difficult to optimize since it has a relatively higher computational cost compared to standard single-objective BO acquisition functions (such as expected improvement) when evaluated using box decomposition. However, one can more efficiently compute EHVI and its gradients exactly (up to a Monte Carlo integration error) using the inclusion-exclusion principle, making the solution of (10) tractable.

[0046] An illustrative summary of the complete hardware-software co-design framework is provided in FIG.3. The overall framework comprises an inner learning procedure that represents a templated workflow to design a signal CoC policy 50. This resides internally at 52 in the right side of the figure. Each run is represented by the path of 52 to 54, with the outer loop that includes the BO 56.

[0047] A particular challenge is the fact that ξ may consist of ordinal and categorical variables. Ordinal variables are those with some type of natural ordering such as continuous (e.g., weight parameters in the control policy) and integer variables (e.g., the number of nodes / layers in the DNN policy). Categorical variables, on the other hand, are best described as a collection of unordered categories such as the choice of parallelization option in the code generation tool. The standard way for dealing with categorical variables in GP models is to apply one-hot encoding (i.e., converts a c-category variable into c new binary variables). The main challenge with one-hot encoding is that it can lead to a large increase in the dimensionality of the search space, as well as complicate the acquisition optimization process. As such, the solution may pursue a mixed kernel function approach that combinesseparate kernels for the ordinal and categorical variables. For every element of J, the discussion focuses on independent kernels of the following form.where ξcatand ξorddenote the categorical and ordinal components of ξ, respectively, and ^^^^^1^^^^^^^^^^^, ^^^^^2^^^^^^^^^^^, ^^^^^1^^^^^^^^^^^, and ^^^^^2^^^^^^^^^^^are kernels associated with the categorical and ordinal variables, each with their own set of hyperparameters. The ^^^^^1^^^^^^^^^^^and ^^^^^1^^^^^^^^^^^kernels are associated with a product in (14) and, thus, can capture the joint impact of both types of variables. The ^^^^^2^^^^^^^^^^^and ^^^^^2^^^^^^^^^^^kernels are associated with a summation in (14), capturing independent impacts of each type of variable.

[0048] Based on the idea of Hamming distances, the categorical kernel is selected as followswhere d(x, y) is meant to represent the distance between categories (equal to 0 if x = y and 1 otherwise) and νiis a variance hyperparameter related to the magnitude of the function and l is a lengthscale hyperparameter related to how fast the function can vary with distance. For the ordinal variables, one embodiment focused on a Matérn-5 / 2 kernel that have similar variance and lengthscale hyperparameters, though this choice could be replaced with any other established kernel.

[0049] As a first look into the complexity of the embedded control problem, one examines the benefits offered by using deep learning as a bridge between hardware and software. A major consideration in embedded control is whether the proposed control policy can be implemented on hardware, which is related to the hardware constraint (1f) and can be a factor of the online computational complexity of the control policy.

[0050] The gap between MPC implementations and DNN implementations can be significantly widened in specialized contexts, e.g., in cold plasma bioprocessing / medicine, where the dynamics are nonlinear, and safety considerations are paramount.

[0051] To illustrate the complete proposed design framework illustrated in FIG.3, the process investigated the CoC design for cold atmospheric plasma (CAP) devices. CAPs have recently found promising use in a variety of applications, including (bio)materials processing and plasma medicine. CAPs, a low temperature (partially) ionized gas, can be generated by applying electric field to a noble gas, typically argon or helium. The synergistic effects of CAPs, including the generation of reactive chemical species and ions, ultraviolet radiation, low-level electric fields, and thermal effects, are posited to induce therapeutic and practical outcomes. CAP devices, such as atmospheric pressure plasma jets (APPJs), can facilitate direct CAP treatments by providing a portable, point-of-use solution to deliver plasma effects in a directed manner. However, APPJs pose unique challenges in control and rely on cutting edge control formulations, most of which have no unified embedded techniques.

[0052] The embodiments here use a kHz-excited APPJ in helium that consists of a copper ring electrode wrapped around a quartz tube, which serves as a dielectric barrier and the gas flow channel. A schematic of the APPJ 70 is shown in FIG.4. Helium gas flows through the tube 72, and plasma ignition is achieved by applying a high-frequency, alternative current (AC) voltage 76 to the copper electrode. The generated plasma is directed out of the tube 72 onto a grounded, metal plate 88 having a glass cover 86 at a distance of 4 mm below the tip of the tube.

[0053] In this testbed, the applied power P in Watts and helium flow rate q in standard liters per minute (s.l.m.) are the manipulated inputs. The maximum surface temperature T in degrees Celsius and total optical intensity of the plasma I in arbitrary units (arb. units) at the plasma-surface incident point are the measured outputs made available every 0.5 s. In theexperimental testbed, the applied power is implemented using an embedded proportional- integral (PI) controller on a microcontroller (Arduino UNO) 82 that manipulates the applied voltage.

[0054] Furthermore, the applied voltage signal is created by generating a sinusoidal waveform at a specified frequency using a function generator 80; in one embodiment this may comprise an integrated circuit, such as an XR-2602CP. This signal is amplified using an amplifier 76, in one embodiment the amplifier may comprise a TREK 10 / 40A-HS before being sent to the copper electrode 74.

[0055] Surface temperature is measured through a radiometric infrared thermal camera 90 such as a Lepton FLIR 3, and optical plasma intensity is measured via a fiber optic cable connected to an optical emission spectrometer 92. In one embodiment, the spectrometer may comprise an Ocean Optics USB2000+. Data acquisition is implemented and managed via USB connection to any standard CPU 84, in one embodiment using Python.

[0056] In one specific embodiment, DNN-based CoC policies were implemented on a field programmable gate array (FPGA, the programmable logic side of a Zybo Z7, XC7Z020- 1CLG400C). The programming files for the FPGA were generated automatically using MathWorks HDL Coder (included with MATLAB R2021a) and Xilinx Vivado 2020.1. FPGA-in-the-loop simulations and experiments were facilitated by MATLAB on a standard laptop CPU, rather than the Zybo Z7’s on-board processor. In summary the system comprises an atmospheric pressure plasma jet positioned to direct the plasma at a surface, a controller communicatively coupled to the computing device to receive selected controller parameters and generate process inputs to the jet, and a computing device configured to receive measurements from the surface and the plasma jet, and to select the controller parameters. One or more processors in the computing device then iteratively operates a DNN, the DNNconfigured to determine a control policy for the controller, and then embeds the control policy by programming the controller.

[0057] The below discussion focuses on the plasma jet, but one should understand that it serves as an example of more widely applicable process. Generally, one embodiment comprises a method of programming a controller, wherein one or more processors receive measurements from a device under control of the controller. The process then iteratively operates the DNN, the DNN configured to determine a control policy for the controller using measurements of the closed-loop performance until the measurements received indicate acceptable performance. Once the performance has reached the desired level, the control policy is embedded by programming the controller.

[0058] A critical challenge in the development of model-based optimal control policies lies in the modeling of the CAP and its interactions with the target surface. CAPs are notoriously difficult to model since they exhibit nonlinear dynamics that are distributed over multiple lengths and time scales. Modeling difficulty is further exacerbated by the intrinsic variability in the plasma and sensitivity of the APPJ to exogenous disturbances. Moreover, use of theoretical models is ill-suited for real-time control of plasma effects that occur on the millisecond to second timescale. Instead, a common solution is to resort to data-driven modeling of the APPJ.

[0059] Here, using the n4sid function in MATLAB allows identification of a linear, time- invariant model using input-output data of the APPJ. Input-output data were gathered by performing multiple step tests in the inputs u = [^^^^, q]⊤and recording the outputs y = [T, a×I]⊤, where a is a scaling factor to scale the total intensity to the same order of magnitude as surface temperature. Furthermore, the data were centered around nominal operating conditions [Ps, qs]⊤= [1.5 W, 3.5 s.l.m.]⊤and [Ts, Is]⊤= [33.3 °C, 7.7 intensity units]⊤, wherethe superscript s denotes the nominal condition. The model follows the discrete-time state- space form: xt+1= Axt+ But, (18a) yt= Cxt+ Dut, (18b) where t ≥ 0 is the discrete time step, xis the vector of states, u ∈ ℝ^^^^^^^^is the vector of manipulated inputs, y ∈ ℝ^^^^^^^^is the vector of measured outputs, and A, B, C, D are the state- space matrices identified using subspace identification. The state-space model is defined in terms of deviation variables around the nominal operating condition, i.e., y = [(T −Ts), (aI−Is)]⊤and u = [(P −Ps), (q−qs)]⊤. In this case, one may assume an observable canonical form of (18), where C = I and D = 0.

[0060] Plasma treatment not only depends on the current state of the plasma itself, but also on quantification of the delivered plasma effects to a surface. While quantification of plasma effects is generally cumbersome and application dependent, the process may take inspiration from hyperthermia treatments to quantify the delivery of a desired thermal effect (aka a thermal dose). A thermal dose metric is quantified in terms of cumulative equivalent minutes (CEM), which describes the accumulation of thermal effects on a target with respect to a reference temperature. The CEM is described by:where Tref= 43 °C is the reference temperature and δt is the sampling time. This definition of the thermal dose is cumulative, in that plasma effects delivered cannot be removed, and nonlinear due to the exponential dependence on temperature. The control objective is to deliver a desired “dose” of thermal effects given by a target CEM value.

[0061] Since there is no way to remove thermal effects once delivered, one needs to be cautious in the face of uncertainty. Therefore, the process may select scenario-based MPC, or sMPC for short, to design a controller that is robust to uncertainty especially in the presenceof safety-critical constraints. Specifically, sMPC assumes that the system uncertainty can take on finite number of s scenarios at every time step. Whenever the uncertainty is time-varying in the sense that it can take on new values at every time step, the system evolution can be represented by a scenario tree of S = sNunique combinations of uncertainty values where ^^^^ denotes the prediction horizon. The sMPC policy then solves the following minimization problem at every time step:s.t. xk+1,j|t= Axk,j|t+ Buk,j|t+ wk,j|t, (xk,j|t, uk,j|t) ∈ ^^^^ × ^^^^, (21b)∑^^^^^^^^=1 ^�^^^^^^^ ^^^^^^^^|^^^^ = 0, (21c)x0,j|t= xt, (21d)where the subscript ( ∙ )k,j|tdenotes the j-th scenario predicted k steps ahead of the current time t, ωjis the probability of occurrence of the j-th disturbance sequence Wj|t= (w0,j|t, ... wN−1,j|t), ^^^^ ^^^^^^^^^^^^ ^^^^ are the state and input constraints, respectively, that must hold for all uncertainty realizations, L and Lfare the stage and terminal cost functions, respectively, and (21c) enforces the non-anticipativity constraints, which ensure states that branch from the same parent node have the same control input. These constraints can be written in terms of knownmatrices {^�^^^^^^^ }^^^^^^^^=1 that impose structure on the vector of control inputs for the j-th scenario,i.e., Uj|t= (u0,j|t, ... , uN−1,j|t). To limit the exponential growth in the scenario tree, one can use the idea of the robust horizon Nr< N, which stops branching after Nrsteps with Nrusually equal to 2 or 3 (often sufficient in practice to achieve constraint satisfaction).

[0062] In (21), the control objective is given by a terminal cost of Lf(xN) = (CEMsp− CEMN|t)2where CEMspis the desired CEM setpoint value. The process uses a predictionhorizon of N = 5 and a robust horizon of Nr= 2. The discussion now considers a set of 3scenarios at each time step, mainly {(−^�^^^^^^^^^^^^^^^^^^^^^^^, −^�^^^^^^^^^^^^^^^^^^^^^^^), (0, 0), (^�^^^^^^^^^^^^^^^^^^^^^^^, ^�^^^^^^^^^^^^^^^^^^^^^^^)}, whichcorrespond to low, middle, and high values for the uncertainty for a tuning parameter ^�^^^^^^^^^^^^^^^^^^^^^^^. The inputs are constrained by the hardware to satisfy P ∈ [1.5, 5.0] W and q ∈ [1.5, 5.0] s.l.m. The outputs are constrained by the following limits T ∈ [25, 45] °C and I ∈ [0, 80] intensity units. The optimization (21) was formulated using CasADi and solved with IPOPT in a receding-horizon fashion.

[0063] Although sMPC provides an effective robust control strategy for the APPJ, it poses a significant challenge for CoC design. There are no known off-the-shelf embedded implementations of (21). As such, sMPC is an ideal test case for the proposed framework since it provides a useful expert control policy that can be imitated by a DNN on embedded hardware. TABLE I

[0064] Now, the goal is to embed the sMPC policy defined by (21) on specialized hardware, mainly an FPGA (i.e., the programmable logic side of the Zybo Z7). The process may select CoC design parameters at each stage of the proposed framework – it lists several possible parameters in Table I. The embodiments selected the following closed-loop performancemetrics to define the MOO problem (9) that provides the basis for selecting the complete set of software and hardware parameters ξ:where W = (w0, ... ,wT−1) is the set of random uncertainty values over the horizon T, [a]+ = max{a, 0}, and the expectation is taken over W. Here, J1 represents deviation from the desired setpoint of CEMsp= 1.5 min and J2is a temperature constraint violation metric given Tmax= 45 °C.

[0065] One embodiment selected the following CoC design parameters and their bounds: maximum possible uncertainty realization ^�^^^^^^^^^^^^^^^^^^^^^^^∈ [0, 10], number of nodes per hidden layer in the DNN H ∈ [2, 10], and the fixed-point word length when generating code wl ∈ [10, 32].

[0066] As mentioned above, the constraint function g(θA, γ) can be learned prior to any optimization routine. As such, one embodiment pre-selects a range of H and wl to learn a binary classifier ^^�^^(H,wl) → {0, 1}. Data ^^^^g= {(H,wl), g(H,wl)}were generated by creating DNNs with randomly initialized weights and passing them through the code generation process ^^^^^^^^. Feasibility was recorded with each combination of parameters. Using ^^^^g, a neural-network-based classifier was trained using MATLAB’s fitcnet function with an 80 / 20 training / test split.

[0067] While the generation of training data for the hardware feasibility classifier is performed independently of the CoC design optimization, the process of generating the data itself can be costly due to the time required to run the code generation step (i.e., the FPGA synthesis). To reduce the cost of gathering training data, optimization methods like BO can be used to determine the boundaries of feasible / infeasible points, instead of searching over the entire parameter space of ΘAand Γ.

[0068] FIG.5 shows the confusion matrix of the resulting hardware classifier learned from Dn. The confusion matrix shows a reliable estimation of how well the process can predict the feasibility of a particular hardware design, with 0 representing an infeasible control-on-a-chip design, and 1 indicating a feasible design. As shown in FIG.5, the classifier accurately predicts an 85% test accuracy. This hardware classifier can then be used during an optimization routine to estimate which combination of H and wl are feasible. The result is that the process effectively reduces the search space prior to performing full CoC design runs.

[0069] Five replicates of MOBO are used to iteratively find the Pareto frontier according to the two closed-loop performance metrics using a total of 50 iterations. Closed-loop FPGA-in- the-loop simulations of the APPJ are performed in MATLAB. DNNs are trained with feedforwardnet and ns= 2, 000. Again, the choice of nswas done empirically. However, a useful feature of the CoC framework is that the closed-loop validation process is able to “catch” if poor performance is consistently achieved for a variety of design parameterswhich, in turn, may be indicative of a poor training dataset. Since the CoC optimization is performed offline (up until the real system query), one can always increase the size of nsat the cost of longer data generation and training time. Results from the closed-loop simulations are passed into the MOBO framework implemented in Python using Ax. MOBO is compared to random search (RS) using a quasi-random Sobol sampling strategy, which is a common benchmark for derivative-free optimization methods. Each strategy was initialized with one known, successful design and one Sobol-selected design.

[0070] FIG.6 illustrates the observed closed-loop performance metrics over the five replicates of MOBO and RS. In FIG.6, the circles indicate data points gathered from MOBO, while squares indicate data points gathered from RS. Dashed black lines indicate “objective thresholds,” which are used in Ax to constrain the search space to produce values within adesired region-of-interest. The left diagram shows all data encountered in all optimization routines, and the right shows a “zoomed in” version with a truncated x-axis at 120.

[0071] Objective thresholds are chosen such that the closed-loop performance metrics are representative of valuable or practical control policies. In other words, in the case of the thermal dose metric, if the metric exceeds 100 (≈ CEMsp× 70-time steps), then this would mean that the suggested CoC policy will take longer than the maximum treatment time allowed and / or is incapable of achieving the desired thermal dose. In the case of the temperature constraint metric, if the metric exceeds 80 (≈ 1◦C×80-time steps or 10 °C×8-time steps), then the policy is considered too dangerous as it violates the constraint too often or at too high magnitude. Objective thresholds can be chosen since MOBO produces a surrogate model that relates the design parameters to the performance metrics. The posterior model can be used to estimate parameters that will produce performance metrics that are likely to be in the defined region-of-interest. Since the surrogate model in MOBO is probabilistic in nature, CoC design parameters suggested by MOBO may still fall outside of the region-of interest in the initial iterations of MOBO. Note that RS has no notion of the objective threshold since no surrogate model is created based on the previously observed data. As such, RS explores significantly more designs outside of the region-of-interest due to a less constricted parameter space. The left subfigure shows all data encountered, while the right shows a zoomed-in version with a truncated x-axis, and it illustrates how RS encounters significantly more points with a thermal dose metric greater than 100.

[0072] Furthermore, FIG.7 shows the HV (HyperVolume) evolution over the 50 iterations of each method, with 100 being MOBO and 102 being RS. The solid lines indicate the mean over five replicates whereas the shaded region indicates one standard error. Recall that the HV is a measure of the quality of the Pareto frontier. As seen in FIG.7, the two methods begin in a similar fashion at suboptimal Pareto frontier estimations over the first twoiterations, but MOBO quickly diverges within the immediate next few iterations. MOBO’s increase in HV value in few iterations is due to its ability to intelligently explore the design space in search of the optimal trade-off between the two-performance metrics. Meanwhile, RS is a naive approach that explores many options that are not expected to improve the HV, i.e., designs are selected with no knowledge of the outcome. As such, it takes RS many more iterations to reach a HV close to that of a “converged” MOBO.

[0073] Finally, FIG.8 shows closed-loop trajectories of the states related to J1and J2(CEM and T, respectively). FIG.8 illustrates the performances of various CoC designs that are encountered at several snapshots of MOBO. Using FIG.8, one can see the variety of control policies that are encountered, and how the observed Pareto frontier evolves over time. From top to bottom, FIG.8 show snapshots of one replicate of MOBO at iterations 5, 15, and 25. From left to right, the subfigures in the left column are the closed-loop trajectories of CEM for selected designs; the subfigures in the middle column are the closed-loop trajectories of surface temperature for selected designs; and the subfigures in the right column are the observed metrics from the designs encountered up until that iteration. The process selects designs that a practitioner may select based on the application needs. A “utopia” design is determined on the basis of the lowest combination (scaled sum) of performance metric values: ^^^^^^^^ = ^^^^1+ ^^^^2where ^^^^1and ^^^^2are scaled values of Ji. Bounds of the scaling are fixed to [20, 100] for J1and [0, 60] for J2. A “control performance preferred” design is determined on the basis of a weighted combination of J1and J2,where τ = 3. A “constraint satisfaction preferred” design is determined similarly, but with the weights switched, i.e.,Designs are selected in this manner to avoid overly extreme controller designs. Utopia designs are denoted by solid lines in the trajectory plots; control performance preferred designs are denoted by the dashed lines; constraint satisfaction preferred designs are denoted by the dotted lines. At iteration 5, MOBO has selected designs that improve the Pareto frontier such that a utopia design and control performance preferred design are different from the first design. The utopia point provides an indication of design with the most even trade- off between the different metrics and can also partially indicate the quality of the Pareto frontier. At iteration 15, more designs have been explored such that new designs for the more extreme designs can be observed. At iteration 25, several more designs have been explored, but had little value to finding the Pareto optimal points, and as a result, had no significant changes between iteration 15.

[0074] Finally, to show utility in the real system, the process selected three control policy designs from the Pareto frontier generated in simulation. Note, for ease of testing, one embodiment chose to modify H, L, wl, and loop where loop is a categorical option in the code generation process that determines whether or not to parallelize certain matrix computations. MOBO was used offline to search for optimal CoC designs and transfer the designs to real- time experiments. In principle, these optimal design parameters obtained offline are sufficient for the experiments, as they primarily describe the capability of the CoC policy in (i) accurately representing the sMPC policy and (ii) being feasibly implemented on the hardware device.

[0075] FIG.9 shows the observed data from three replicates of MOBO for the new set of design parameters. FIG.9 shows a similar trade-off as in FIG.6 even with different design parameters. Real-time experiments using the CoC design parameters to create embedded control policies for the APPJ were performed in triplicates. Table II describes the controllerparameters used and the closed-loop metrics obtained on the APPJ testbed. From Table II, one can see several trade-offs between control performance, hardware utilization, and constraint satisfaction. A performance dominated CoC design (i.e., Configuration (i)) is typically comprised of a small-scale DNN that can achieve the desired CEM dose quickly at the expense of more constraint violations. A constraint dominated CoC design (i.e., Configuration (ii)) is typically comprised of a larger width DNN with larger fixed point word length. This often leads to a more representative DNN that has fewer constraint violations. Finally, a mixed CoC design (i.e., Configuration (iii)) can offer a balance of the two extremes. Table II

[0076] One issue that may arise in approximating tracking MPC policies is the need to ensure that the approximation error does not jeopardize the setpoint tracking ability of the DNNcontrol policy. Let ^^^^^^^^^^^^^^^^(^^^^, ^^^^) =denote the DNN control policy.Since only a finite amount of data is considered for training along with other factors related to local training and fixed DNN structure, the approximate control policy introduces some level of error:where ^^^^ is a user-specified reference in the tracking MPC formulation and ^^^^^^^^^^^^^^^^ (^^^^, ^^^^) denotesthe original tracking MPC policy. Therefore, additional modifications are needed to ensure convergence to the desired target condition in the face of this error. This is particularly important for setpoint tracking problems where one expects the system to operate near steady-state for the majority of time.

[0077] The basis for the proposed correction factor is the following target trackingoptimization problem ℙ^^^^(^^^^):s.t. ^^^^^^^^ = ^^^^(^^^^^^^^, ^^^^^^^^), (24^^^^)^^^^^^^^ = ℎ(^^^^^^^^), (24^^^^)(^^^^^^^^, ^^^^^^^^)^^^^ ^^^^ × ^^^^, (24^^^^)where ls : → ℝ is the steady-state cost function that is assumed to guaranteethat ℙ^^^^(^^^^) has a unique solution for every reference ^^^^ ^^^^ ℝ^^^^^^^^ . Let (^^^^∗^^^^(^^^^) , ^^^^∗^^^^ (^^^^), ^^^^∗^^^^ (^^^^))denote the solution to the tracking optimization (24). The proposed off-set free DNN-based NMPC policy then takes the form ^^^^^^^^^^^^(^^^^, ^^^^) = ^^^^^^^^^^^^^^^^ (^^^^, ^^^^) + (^^^^∗^^^^(^^^^) − ^^^^^^^^^^^^^^^^(^^^^∗^^^^(^^^^), ^^^^)), (25)whose properties are summarized next.

[0078] Assume ^^^^ × ^^^^ contains at least one feasible steady state and the offset cost LO(ε)satisfies the properties of an exact penalty function for (24d), wherein LO(ε) contains some penalty weight ρ that is greater than the dual norm of the optimal Lagrange multiplier vector corresponding to the optimal solution to (24) without the softened constraint (24c). Then, theoff-set free DNN NMPC policy (25) exists for any reference ^^^^ ∈, and guarantees that aclosed-loop equilibrium point (x∞,u∞) exists such that r = h(x∞) for any feasible reference r.

[0079] This follows from ℙ^^^^(^^^^) that is feasible for all ^^^^ ∈ ℝ^^^^^^^^ when (xs, us) ∈ ^^^^ × ^^^^ for atleast one point xs= f(xs, us). This is due to the softened constraint r = h(xs) through the inclusion of ys, i.e., r = h(xs) + ε for any ε = r- ys∈ ℝ. When LOis an exact penalty functionfor this constraint, a feasible r impliesOne canshow that the closed-loop system ^^^^+ = ^^^^(^^^^, ^^^^^^^^^^^^(^^^^, ^^^^)) preserves this equilibrium point since^^^^^^^^^^^^(^^^^∗^^^^(^^^^), ^^^^) = ^^^^^^^^^^^^^^^^(^^^^∗^^^^(^^^^), ^^^^) + ^^^^∗^^^^(^^^^) − ^^^^^^^^^^^^^^^^(^^^^∗^^^^(^^^^), ^^^^) = ^^^^∗^^^^(^^^^).

[0080] This result indicates that the offset-free equilibrium point exists for the proposed DNN-based controller; however, it does not prove that the controller makes the system converge to equilibrium. In fact, it is quite likely that a poorly-trained DNN may lead to closed-loop instability. Therefore, it is still necessary to perform some level of stability analysis and / or performance validation before deploying the DNN controller in practice.

[0081] It is interesting to note that, whenever the approximation error ^approx= 0, (16) reducesto ^^^^^^^^^^^^(^^^^, ^^^^) = ^^^^^^^^^^^^^^^^(^^^^, ^^^^) = ^^^^^^^^^^^^^^^^(^^^^, ^^^^) since the DNN matches a steady-state condition andthe proposed correction term goes to zero. This provides insight into the correction term that effectively corrects for bias around the equilibrium condition. That is, the closer the DNN approximates the original MPC policy, the more likely the closed-loop system, such as the cold plasma get discussed above, to be attracted near the equilibrium manifold. These fairly small errors can be overcome as no other equilibrium with the same steady state input exists.

[0082] The embodiments here present an end-to-end control-on-a-chip (CoC) design framework for the implementation of arbitrary control policies on arbitrary hardware. The embodiments used deep learning as a unifying template that connects the hardware and software aspects of embedded control design. Furthermore, the embodiments may include a Bayesian optimization framework for CoC design that can account for the multi-objective nature of the control design problem, categorical design spaces, and minimal interactions with the expensive design process. The discussion demonstrated the proposed CoC designframework for cold atmospheric plasma processing of biomaterials in closed-loop simulation studies and real-time experiments. The framework was able to efficiently and systematically determine trade-offs in the CoC design process, resulting in adequate estimation of the Pareto frontier in only a few design iterations.

[0083] Additionally, this written description makes reference to particular features. It is to be understood that the disclosure in this specification includes all possible combinations of those particular features. For example, where a particular feature is disclosed in the context of a particular aspect, that feature can also be used, to the extent possible, in the context of other aspects.

[0084] Also, when reference is made in this application to a method having two or more defined steps or operations, the defined steps or operations can be carried out in any order or simultaneously, unless the context excludes those possibilities.

[0085] All features disclosed in the specification, including the claims, abstract, and drawings, and all the steps in any method or process disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive. Each feature disclosed in the specification, including the claims, abstract, and drawings, can be replaced by alternative features serving the same, equivalent, or similar purpose, unless expressly stated otherwise.

[0086] Although specific aspects of this disclosure have been illustrated and described for purposes of illustration, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. Accordingly, the invention should not be limited except as by the appended claims.

Claims

WHAT IS CLAIMED IS:

1. A system comprising: a device capable of performing a physical operation, the physical operation having quantifiable measures of its controlled performance; a controller communicatively coupled to a computing device to receive selected controller parameters and generate control inputs for the device; and the computer device having one or more processors configured to execute code to cause the one or more processors to: iteratively receive sets of measurements of the quantifiable performance measures when the device is operating under control of the controller; operate a deep neural network (DNN) for each set of the measurements until performance of the controller is determined to be acceptable, the DNN configured to determine a control policy for the controller; and embed the control policy by programming the controller when the performance is acceptable.

2. The system as claimed in claim 1, wherein the code that causes the one or more processors to operate a deep learning network (DNN) for each set of measurements comprises code that causes the one or more processors to perform Bayesian optimization on the DNN for that set until the performance of the controller is determined to be acceptable.

3. The system as claimed in claim 1, wherein: the device capable of performing a physical operation comprises an atmospheric pressure plasma jet positioned to direct the plasma at an interface.

4. The system as claimed in claim 3, wherein the quantifiable performance measures comprise cumulative equivalent minutes (CEM) and surface temperature.

5. A method for automatically designing a controller, comprising:designing a physics-informed control policy; testing the physics-informed control policy using a high-level programming language; creating a policy representation using a deep neural network; correcting any offset error in control performance arising from errors of a deep neural network approximation of a deep neural network policy; iteratively operating and evaluating the deep neural network until performance meets an acceptable performance; implementing the deep neural network into hardware; and evaluating performance of the hardware against a desired performance.

6. The method as claimed in claim 5, wherein implementing the deep neural network comprises an approximation of an implicit optimization-based control method.

7. The method as claimed in claim 6, wherein the optimization control method comprises a model predictive control strategy.

8. The method as claimed in claim 7, wherein the model predictive control strategy has one or more of a learning-based variant, a robust variant, and a stochastic variant.

9. The method as claimed in claim 5, wherein designing the physics-informed control policy includes a mathematical description of system dynamics for a system in which the controller will operate, constraints on operation of the system, and control objectives.

10. The method as claimed in claim 5, wherein correcting for any offset error comprises one or more of an optimization error estimation or an approximation thereof.

11. The method as claimed in claim 5, wherein testing the physics-informed control policy comprises using a derivative-free optimization.

12. The method as claimed in claim 11, wherein the derivative-free optimization comprises one of either Bayesian Optimization or Sobol Random Sampling or Reinforcement Learning.

13. A method of correcting errors in a deep neural network control policy such as a policy obtained by approximating a model predictive control policy, comprising: solving a tracking optimization problem online that yields the optimal steady-state state and input variables corresponding to a user-specified reference signal; and using the optimal steady-state state and input variables to correct the approximation error of the deep neural network control policy in real-time. This method enables offset-free tracking of the reference signal despite approximation errors.

Images