Neural network controller

Abstract

The invention relates to a method of generating a dynamic system controller for controlling a dynamic system, the dynamic system controller comprising a neural network. The method comprises: modelling the dynamic system to generate a system model; defining a control challenge training set for evaluating candidate neural networks, the control challenges derived from the system model; generating an optimised neural network from a population of candidate neural networks using a neuroevolution optimiser, wherein the population of candidate neural networks is evaluated using the control challenge training set; validating the optimised network using a validation set of control challenges derived from the system model; deploying the optimised neural network to a hardware test controller for the system; evaluating the performance of the hardware test controller in a hardware test; and, if the hardware test controller fails the hardware test, generating a further optimised neural network and repeating the validating, deploying and evaluating steps; or, if the hardware test controller passes the hardware test, deploying the optimised network to a controller for controlling the system.

Description

Technical Field

[0001] This disclosure relates to a controller for controlling dynamic systems, particularly suitable for industrial systems. Aspects of the invention relate to a method for generating a controller, a neural network for a controller, and a controller itself. Background Technology

[0002] In almost every industrial process, a process controller is needed to stabilize and optimize process performance.

[0003] Currently, proportional-integral-derivative (PID) controllers are used for process control in numerous industrial systems. PID controllers use a feedback control loop mechanism to control process variables such as pressure, temperature, and flow rate, as shown in the block diagram in Figure 1. The PID controller compares the output process variable with its desired setpoint and calculates the required input to the system based on a set of linear mathematical functions to bring the process variable closer to the desired setpoint. PID controllers have proven highly effective in controlling relatively simple processes involving a controlled return to the setpoint in a single-input / single-output system. These systems often exhibit linear dynamics and low uncertainty, such as certain types of valve control systems or motor control systems.

[0004] However, because PID controllers are based on relatively simple control strategies, which are typically used for tuning or optimizing within a limited operating range, they are not well-suited for efficiently controlling more complex systems. If the process is nonlinear, time-varying, has discontinuous state changes, contains unforeseen disturbances, or has a large number of inputs and outputs, the PID controller may fail to accurately control the process, and control may begin to degrade when the system exceeds its narrow optimization range. To enable PID to be applied to more complex systems and dynamically adjust setpoints in industrial processes, PID can be cascaded (as a cascaded controller) or combined with more advanced methods such as fuzzy logic and model predictive control (MPC) (discussed later). However, when employing cascaded PID or a strategy combining PID with another control method, there is an inherent trade-off between setpoint tracking and system stability. Therefore, more complex systems require advanced process control (APC), which necessitates the use of other control techniques.

[0005] In some complex systems, APC (Advanced Process Control) can be performed using lookup tables for PID controllers, which allow for different setpoints for varying system states. This is also known as gain scheduling. In gain scheduling, a lookup table is created containing a database of PID controllers, each with different constant or "setpoint" settings. Therefore, as the system changes, the PID controllers used to control the system can switch, resulting in state transitions within the system. However, state transitions require the system to detect which PID to use based on current conditions, which can itself be a difficult inference problem. Furthermore, rapidly switching from one PID controller to another can lead to system instability and oscillations. Moreover, gain scheduling is limited to the known range of variations within the system and cannot handle a wide range of uncertainties and nonlinearities. Therefore, the effectiveness of PID lookup tables in controlling complex systems is limited.

[0006] Model predictive control (MPC) is the second option for APC and is currently the most widely used APC method. In MPC, a mathematical model of the process is generated, which can be based on a parametric physical model, differential equations, or a data-driven model. During operation, MPC iteratively computes the optimal output sequence based on measurements of the system state. However, even in applications where MPC works well (typically where very high response frequencies are not required), significant drawbacks exist.

[0007] MPC has low robustness: it is impractical to mathematically incorporate any unexpected changes in the controlled system (such as equipment variations or wear and tear) into the model. If such changes occur, a large portion of the model needs to be redeveloped, or system performance will degrade.

[0008] In the context of controllers, another significant drawback of MPC is its computational intensity (the entire model must be rerun at each process step and continuously throughout the control loop). Therefore, several limitations of using MPC for APC are evident.

[0009] Recently, artificial intelligence and machine learning (AI / ML) technologies have been considered potential control methods for automated control systems (APCs). However, systems requiring APCs, such as power generation systems and chemical engineering systems, typically undertake applications of significant economic value, and such systems often involve personal safety risks. Therefore, a high degree of determinism regarding the system and, consequently, the controller is crucial. Currently, AI / ML networks are not widely used in APCs for several reasons.

[0010] First, AI / ML control methods for APC require massive datasets to train neural networks to ensure satisfactory performance in edge cases. For complex systems requiring APC, the need for datasets covering all possible scenarios necessitates massive amounts of data for optimal training of traditional AI / ML controllers. However, in industrial processes, executing system failures and acquiring data for such scenarios is too disruptive, dangerous, and costly to obtain such data. In fact, if AI / ML controllers need to be able to cope with catastrophic failures, as is the case in industrial processes, thousands of catastrophic failures would be required to provide the data needed to train the network, which is simply not feasible.

[0011] Furthermore, training with large datasets results in massive models with numerous layers and high computational intensity. Therefore, similar to the MPC discussed above, running such networks is costly, and the expense of operating AI / ML networks may deter their use.

[0012] Finally, when AI / ML solutions are used in APC, they often fail to adapt well to real-world environments. This is related to the lack of data, discussed above, covering all possible scenarios for training the AI / ML controller. The AI / ML network may be trained, but when implemented in practice, it may fail because it cannot scale to the real-world environment. Therefore, AI / ML controllers struggle to adapt to process changes, unexpected events, and failure modes. Typically, this means that existing AI / ML controllers are limited to automating under stable conditions and providing advice to operators. For these reasons, currently available AI / ML controllers are not suitable for APC and have not yet been widely adopted in this application.

[0013] As mentioned above, while Automation Process Control (APC) is crucial for ensuring the smooth operation of various industrial processes, a robust, cost-effective, and computationally efficient automation method is currently lacking. Furthermore, many APC methods currently used in industry are not properly optimized or calibrated. Optimization is extremely important for achieving significant benefits, including increased output, improved productivity, and reduced energy consumption. Therefore, there is a strong need for an APC method capable of optimization for specific industrial processes.

[0014] Therefore, the object of the present invention is to provide a controller that is not limited by the above and can be effectively adapted to practical applications for use in industrial systems. Summary of the Invention

[0015] According to a first aspect of the invention, a method for generating a dynamic system controller for controlling a dynamic system is provided, the dynamic system controller including a neural network, the method comprising: modeling the dynamic system to generate a system model; defining a control challenge training set for evaluating candidate neural networks, the control challenge derived from the system model; generating an optimized neural network from a group of candidate neural networks using a neuroevolutionary optimizer, wherein the group of candidate neural networks is evaluated using the control challenge training set; validating the optimized network using a control challenge validation set derived from the system model; deploying the optimized neural network to a hardware test controller for the system; evaluating the performance of the hardware test controller in hardware testing; and, if the hardware test controller fails the hardware test, generating a further optimized neural network and repeating the validation, deployment, and evaluation steps; or, if the hardware test controller passes the hardware test, deploying the optimized network to a controller for controlling the system.

[0016] This invention provides a method for generating a dynamic system controller for a dynamic system, wherein the system is first modeled to define a system model. Then, a control challenge training set is derived from the system model and used to evaluate a group of candidate neural networks.

[0017] This system model provides an environment in which neural networks can be trained and a series of test / training scenarios can be created that would be impractical or prohibitively costly to provide using real dynamic systems. A gradient-free optimizer in the form of a neuroevolutionary optimizer is used to evaluate candidate neural networks to generate optimized networks. The optimized neural networks are then validated against a control challenge validation set to detect overfitting and test the network's performance under more extreme challenges, including challenges exceeding the expected difficulty for controlling real dynamic systems.

[0018] After validation, the optimized neural network is deployed to a hardware test controller for hardware testing. If the optimized neural network fails the hardware test, a further optimized neural network can be generated; for example, the system model can be adjusted based on the reasons for the hardware test failure. If the optimized neural network passes the hardware test, it can be deployed to a dynamic system controller used to control dynamic systems.

[0019] It should be noted that the generation of the optimized neural network may also include evaluation on a control challenge validation set to detect overfitting or undergeneralization. After validation, the method can proceed to the aforementioned validation process, which tests the extrapolation behavior of the optimized network and its performance in edge cases.

[0020] The dynamic system may include actuators for driving joints in a robotic arm, feedback control loop mechanisms for controlling pressure, temperature, or flow variables, motors in an unmanned aerial vehicle, or any system that can conventionally use a PID controller or an MPC controller.

[0021] Typically, generating further optimized neural networks may involve updating the system model and / or defining a further set of control challenges. In this way, the reasons why previously optimized neural networks failed hardware testing can be incorporated into the system model or curriculum control challenges of the next optimized neural network. This hardware testing may include testing the dynamic system controller under a series of operating scenarios, and the further set of control challenges can be defined based on the operating scenarios that caused the hardware tests to fail.

[0022] A neural network may include neurons and synapses configured to form a directed graph structure, wherein at least some of the synapses in the neural network are configured to have strengths that can be changed during network operation. The graph structure of a neural network allows for the encoding of circuits and arbitrary directed graphs with fewer neural structures than conventional neural networks. This enables the neural network to learn and process more complex relationships with fewer computational units.

[0023] The control challenge training set can include a series of control challenges with progressively increasing complexity. Training courses can be defined to guide the evolution of the neural network.

[0024] The control challenge validation set can include control challenges that exceed the expected difficulty of controlling the dynamic system. Neural networks can accept challenges in more extreme scenarios that may exceed the expected real-world challenges that the controller may need to deal with, such as the failure of multiple rotors in an unmanned aerial vehicle or a robot leg encountering an unexpected object on the ground.

[0025] A dynamic system controller can be configured to receive the current system variables of a dynamic system and control the control variables of that dynamic system to setpoints by outputting control signals. The dynamic system controller can also be configured to receive further constraints, such as state-space regions that the controller should avoid or must not enter.

[0026] The dynamic system may include actuators, and the dynamic system controller may be configured to output control signals to drive the actuators toward a set value. The dynamic system may also control multiple actuators, and the controller may be configured to output control signals to drive the multiple actuators.

[0027] In one embodiment, a robot is provided, including a jointed robotic limb configured to be driven by a motor, wherein a dynamic system controller generated according to a first aspect of the invention is configured to control the motor to drive the joint.

[0028] According to a second aspect of the present invention, a method for controlling a dynamic system using a dynamic system controller generated according to a first aspect of the present invention is provided.

[0029] According to a third aspect of the present invention, a dynamic system controller generated according to the first aspect of the present invention is provided for use in controlling dynamic systems.

[0030] According to a fourth aspect of the present invention, a dynamic system controller generated according to a first aspect of the present invention is provided. The dynamic system controller may include an input for receiving setpoints and current system values ​​of system parameters, and an output for outputting a control signal to control the dynamic system to the setpoints, wherein an optimized neural network is configured to generate the control signal based on the received setpoints and current system values.

[0031] According to a fifth aspect of the present invention, a neural network for a dynamic system controller of a dynamic system is provided, the neural network comprising: a plurality of neurons n and synapse e It is configured to form a directed graph structure, wherein at least some synapses in the neural network are configured to have synaptic strengths that can be changed during network operation.

[0032] Each neuron n During the execution of a clock instance, an activation value can be generated based on the input activation values ​​computed at the previous clock instance. Each neuron can be configured to aggregate the input activation values ​​received from the input neurons at execution time to compute the activation value, wherein the input activation values ​​are weighted by the strength of the connecting synapses.

[0033] The synaptic strength of at least some synapses can be configured to change based on previous neuron activation values ​​during network operation.

[0034] synapse e It can include a set of processor synapses and a set of regulator synapses, each processor synapse configured to contribute to the activation of neurons, and each regulator synapse configured to contribute to the instantaneous learning rate of neurons. This allows the network to adjust its function in the form of unsupervised learning during operation, thereby adapting to unexpected situations.

[0035] Neurons may include activation functions and instantaneous learning rates, and synapses may include learning rules encoded as functions of current synaptic strength, presynaptic and postsynaptic activations, and the learning rate. The activation function, instantaneous learning rate, and learning rules are used to calculate the strength of each synapse.

[0036] Networks can encode multi-directed graph structures. In other words, network connections can be directed, can be parallel (multiple edges can connect two nodes), and can form cycles or loops (edges from a node to itself).

[0037] The initial synaptic strength can be configured to be quantized. Each neuron may include an activation function containing gain parameters.

[0038] According to a sixth aspect of the present invention, a dynamic system controller for controlling a dynamic system is provided, comprising a neural network according to a fifth aspect of the present invention.

[0039] The dynamic system controller may include an input for receiving dynamic system operating data and an output for outputting control signals to control the dynamic system, wherein the neural network may be configured to receive the dynamic system operating data as its input, and the controller may be configured to determine the control signals based on the output of the neural network.

[0040] According to a seventh aspect of the invention, a controller for controlling a dynamic system is provided, the controller comprising: an input for receiving set values ​​of control variables of the dynamic system and current system variables of the dynamic system; a processing means for calculating a control signal to control the dynamic system to the set values; and an output for outputting the control signal, wherein the processing means comprises a trained neural network having neurons and synapses connected in a directed graph structure, and at least some of the synapses in the neural network are configured to have synaptic strengths that can be changed during network operation.

[0041] According to an eighth aspect of the invention, a method for generating a dynamic system controller for controlling a dynamic system is provided, the dynamic system controller including a neural network, the method comprising: modeling the dynamic system to generate a system model; defining a control challenge training set for evaluating candidate neural networks, the control challenge derived from the system model; generating an optimized neural network from a group of candidate neural networks using a gradient-free optimizer, wherein the group of candidate neural networks is evaluated using the control challenge training set; validating the optimized neural network using a control challenge validation set derived from the system model; deploying the optimized neural network to a hardware test controller for the system; evaluating the performance of the hardware test controller in hardware testing; and, if the hardware test controller fails the hardware test, generating a further optimized neural network and repeating the validation, deployment, and evaluation steps; or, if the hardware test controller passes the hardware test, deploying the optimized network to the dynamic system controller for controlling the dynamic system.

[0042] The preferred features of the above-described aspects of the present invention can be applied to other aspects of the present invention. Attached Figure Description

[0043] One or more embodiments of the present invention will now be described by way of example only, with reference to the accompanying drawings, wherein: Figure 1 is a block diagram of a PID controller; Figure 2 This is a block diagram providing an overview of a neural controller according to an embodiment of the present invention; Figure 3 yes Figure 2 A flowchart illustrating the development and training methods of a neural controller; Figure 4 This is a schematic diagram of clock control for a neural network according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating an example of a neural network in an arbitrary directed graph structure; Figure 6 This is a schematic diagram of an example of a neural network with processor synapses and modulator synapses; Figure 7 This is a schematic diagram of an example of a computational neuron according to an embodiment of the present invention, illustrating neuron and synaptic properties; Figure 8 This is a schematic diagram illustrating an example of a limiting function according to an embodiment of the present invention; Figure 9 This is for subsequent creation according to embodiments of the present invention. Figure 2 A flowchart of the neural network development and training method for a neural controller; Figure 10 yes Figure 9 The method combines a feedback flowchart; Figure 11 It is for creating according to embodiments of the present invention Figure 2 A block diagram of the neural controller system; Figure 12 An example of an easing function that can be used to construct a course development waveform according to an embodiment of the present invention is shown; Figure 13 It shows the use of Figure 12 Example waveforms created by the easing function; Figure 14 This is a schematic diagram showing the hardware setup used to perform a swinging and lifting experiment on a robotic arm; Figure 15 This illustrates the use of a neural controller according to an embodiment of the invention to control Figure 14 The results of the experiment performed by the robotic arm are shown below. Figure 16 This demonstrates the use of a PID controller for control. Figure 14 The results of the experiment performed by the robotic arm are shown below. Figure 17 This is a schematic diagram showing the hardware setup used for conducting experiments on the control of multi-joint robot limbs; Figure 18 This illustrates the use of a neural controller according to an embodiment of the present invention to control Figure 17 The results of the experiment on the robot limbs are shown. Figure 19 This is a schematic diagram illustrating the hardware setup for experimental control of a multi-jointed robot limb in a free-swinging experiment; and Figure 20 This illustrates the use of a neural controller according to an embodiment of the present invention to control Figure 19 The results of the robot limbs used in the experiment are shown. Detailed Implementation

[0044] Figure 2 The block diagram illustrates an overview of a neural controller 20 (also referred to herein as a "dynamic system controller" or "controller") for a dynamic system 26 according to an embodiment of the present invention, which operates to control the dynamic system 26. As shown, input variable 22 is input to neural controller 20 from input terminal 23. Neural controller 20 includes an optimized neural network (as described herein) that subsequently generates control signals in the form of control parameters 24. Figure 2 In this configuration, the optimized neural network resides within processor 21. Control parameters 24 are then output from output 25 to control process 26 (i.e., the dynamic system), where they regulate process control. Input variables 22 may be, for example, setpoints that the control variables of the control process should maintain, process limits, or any additional contextual data. Output variables 28 (also referred to herein as "current system values" or "current system variables") relating to the state of system 26 are output from dynamic system 26 and fed back to neural controller 20 via input 23. This feedback is used to update control parameters 24, as determined by neural controller 20, during closed-loop control.

[0045] Figure 3 The flowchart in the document describes Figure 2 The method 30 for developing and training the neural controller 20 is as follows: First, in step 32, the neural network is optimized using a gradient-free optimizer (the method of creating and training this network will be discussed in detail later). Then, in step 34, the optimized neural network is compiled into a netlist, which defines the network according to a microarchitecture. This microarchitecture describes the types of neural networks that can be created and defines the allowed structure and execution behavior of the network. The netlist describes the neural network that has been optimized for the application. The netlist is an intermediate representation of the network architecture, including a list of input neurons, computational neurons, output neurons, and synapses. For example, the netlist can be a compressed JSON encoding of the network structure and parameters. Once the netlist is created, the neural controller 20 is generated in step 36 using the netlist and a network runner, where the runner is a component that executes the netlist on hardware.

[0046] As described above, creating the neural controller 20 involves using an optimizer to optimize and train the neural network. Typically, the optimizer utilizes functions...f ( x ) =y Then determine to make y Maximize or minimize x Value. When this function is applied to optimize a neural network, x It is a specific neural network. f This is course evaluation, used to assess the "quality" of the generated networks (discussed in more detail later), while y It is the calculated fitness, which corresponds to the network's performance (the higher the fitness, the better the performance of the neural network).

[0047] Various methods can be used to optimize neural networks, with gradient descent and backpropagation being the most common. In gradient descent, f The gradient of a gradient is often noisy or has multiple local optima, so gradient descent optimizers may prematurely converge to local optima rather than global optima. In traditional deep learning methods, data scientists estimate a fixed set of neural network architectures (based on literature and expected results) and then apply automatic backpropagation to find the network weights. This approach often results in large and inefficient networks (very deep layers), requiring a large number of training samples and computational power, and has poor scalability. Furthermore, deep learning methods struggle to adapt to differences between training data and real-world examples, resulting in a "reality gap." This "reality gap" leads to poor predictive performance of deep learning networks due to reasons including input variables the system has never seen, underfitting or overfitting, or encountering unforeseen system changes and situations. Therefore, different optimization methods may be more suitable. In one embodiment of the invention, a gradient-free optimizer is used to optimize the neural network.

[0048] Several gradient-free optimizers exist, such as those using neuroevolution. Neuroevolution is a machine learning technique that utilizes evolutionary algorithms to construct artificial neural networks. Neuroevolution is particularly suitable for control applications because networks generated using neuroevolution are extremely efficient: the network structure is more compact, resulting in higher computational efficiency. Neuroevolutionary optimizers are also better at finding the global maximum of the network and are less likely to converge prematurely to local maxima compared to other optimization methods. While neuroevolutionary optimizers are particularly suitable for this application, any other suitable optimizer, such as a random search optimizer, can also be used.

[0049] return Figure 3 A microarchitecture is a mathematical description that defines the set of possible neural networks that can be generated, and includes mathematical rules governing the network's neurons and synapses. A microarchitecture can define one or more properties of the network that will form the controller 20, which will be discussed below. Figure 4 Let's discuss this in detail.

[0050] First, the microarchitecture comprises neurons (nodes) 40 and synapses (edges) 42, where nodes 40 have interconnected synaptic links. Nodes 40 and synapses 42 together form a directed graph. During network execution, each neuron 40 aggregates activation values ​​from its input neurons, where the input activation values ​​are weighted by the strength of the connecting synapses to calculate the neuron's own activation value.

[0051] Another characteristic of the microarchitecture definition is that synaptic strength is plastic in the generated neural network. Plastic synapses allow synaptic strength to be altered based on previous neuron activation values ​​during network operation. The degree of synaptic plasticity (i.e., the learning rate, which relates to the extent to which synaptic strength can be changed during network execution) can be modulated by the neurons 40 connected to the synapses 42, thereby enabling self-correction and mode switching within the network. Essentially, the network adjusts synaptic weights during execution as data flows through it. It should be understood that in the generated neural network, plastic and fixed synapses will coexist, and the proportion of these types of synapses 42 will vary depending on the specific network required to control the application.

[0052] The microarchitecture also specifies that the neural network is configured such that each computational neuron 40 generates the activation value computed in the previous execution, i.e., the activation value at time t-1, at each execution (clockwise) of the network. Figure 4 As shown. This network property is referred to in this paper as "clock control". In a traditional network, neuron 40 generates an activation value at time t for the current execution. Clock control allows the network to encode arbitrary (potentially cyclic) directed graphs, such as... Figure 5 As shown, and with multidirected graphs (edges having the same end node), this means that network connections are directed, can be parallel (there can be multiple edges between two nodes), and can form cycles or loops (edges from a node to itself). The ability to encode loops and arbitrary directed graphs means that certain relationships can be encoded with fewer neural structures, and correspondingly, the network can learn complex functions with fewer computational units. Clock control brings benefits because without this property, neuron activation values ​​would be impossible to compute due to infinite recursion.

[0053] The microarchitecture can also define synapse 42 as excitatory (with positive weights) or inhibitory (with negative weights), and in some embodiments, the initial synaptic strength is quantized, but this depends on the optimizer used.

[0054] Neural Network Properties Examples of neural network properties that can be incorporated into the controller 20 according to embodiments of the present invention will now be described.

[0055] A network defined by a microarchitecture is a graph network, which can be represented as ,in: - N is a set of 40 neurons; - S is a collection of synapses; - source: It is a mapping from each synapse to its input neuron; - target: It is a mapping from each synapse to its output neuron.

[0056] Neurons in a network 40 and synapse 42 has specific properties. First, regarding neurons 40, each neuron... 40 has several attributes, examples of which are listed below: - Gain (Setting the gain to be greater than 1 can counteract the tendency for neuron activation values ​​to decrease as the network deepens.) - Base activation value (or bias) 52, which relates to the activation value exhibited by the neuron per clock cycle when the neuron has no non-zero processor input; - Base learning rate, also known as maximum learning rate ,in This represents the maximum extent of plastic synaptic renewal; - Activation value at time t ; - The learning rate at time t .

[0057] It should be understood that in other embodiments, the specific values ​​of the properties of neuron 40 may vary, and some properties of neuron 40 may not be defined by the microarchitecture.

[0058] Synapses in networks 42, each synapse 42 has the following characteristics: - Learning Rules ; - Type: Excitatory or inhibitory; - Intensity (Quantifiable or unquantifiable); - Weight The weight of excitatory synapses The weight of inhibitory synapses .

[0059] At least some synapses 42 in the network can be plastic and their plasticity can be modulated. This modulation is a form of unsupervised learning and enables the network to easily adapt to unexpected situations or new data. A characteristic of network architectures that allow for such plasticity modulation is the set of neurons and synapses... Divided into processor synapse sets and modulator synapse set ,like Figure 6 As shown. Processor synapse 42a contributes to neuronal activation, while modulator synapses... 42b contributes to the instantaneous learning rate of neuron 40. This refers to the magnitude of the change in synaptic strength at time t. It is advantageous for each neuron 40 in the network to have both processor synapses and modulator synapses, as this allows the signal to contribute to the network's output (via processor synapse 42a), or to influence internal network parameters (via modulator synapse 42b), or both simultaneously.

[0060] Figure 7 The diagram illustrates computational neuron 40, showing modulator synapse 42b and processor synapse 42a, as well as other neuronal and synaptic properties. It should be noted that... Figure 7 In the diagram, "a" represents the activation function, which is used to calculate the neuron's activation value. Furthermore, "0.5" (indicated by reference numeral 54) is the base adjustment value. The effect of the base adjustment value 54 on the modulator synapse 42b is equivalent to the effect of the base activation value "b" (indicated by reference numeral 52) on the processor synapse 42a; that is, the base modulation value 54 contributes to the regulation regardless of other regulator inputs. However, unlike the base activation value 52, the base adjustment value 54 is set as a constant, which is 0.5 in this example.

[0061] The modulator synapse 42b enables neuron 40 to modulate its plasticity. This modulation is based on neuronal activation values. Instantaneous learning rate of neurons 40 and the learning rules encoded on neuron 40 Using the above parameters, the plasticity of synapse 42 at each time point after initialization (t=0) can be determined. .

[0062] For a given neuron 40, at the moment t The activation value of +1 is given by the following formula:

[0063] in, It is input to the neuron The processor synapse 42a of 40, This is the activation value of the input processor synapse 42a. This activation value follows a limiting function, defined as follows:

[0064] Figure 8 An example of the limiting function 55 is shown. Although the activation function is described as following the limiting function, it should be understood that other nonlinear functions may be used in other embodiments. .

[0065] It should be understood that neuron activation (Formula 1) is similar to the classic formula for neuron activation in a perceptron (a well-known rule used in machine learning to calculate activation values ​​of deep learning networks), but there are two differences: a gain parameter is added, and the nonlinear activation function is a limiting function.

[0066] Neuron 40 at time 1 t +1 instantaneous learning rate Similar to the pattern of neuron activation functions (i.e., the instantaneous learning rate function also includes a multiplicative accumulation operation followed by a nonlinear activation function), it is given by the following equation:

[0067] in, For input to neurons The set of regulator synapses 42b of 40, This is the activation value for the input regulator synapse 42b. The constant 0.5 refers to the base regulation 54 of the regulator synapse 42b.

[0068] Each synapse 42 in the network is encoded with a learning rule. Each learning rule is encoded as the current synaptic strength. s Presynaptic activation value α and postsynaptic activation value β and instantaneous learning rate r The function. The learning rule encoded on each synapse describes the synaptic strength. This is based on how the performance of the previous network changes over time. The following are examples of learning rules, but many other rules can also be used:

[0069] The activation function and instantaneous learning rate of neuron 40, along with the learning rules encoded on the synapses, can be used to calculate the strength of each synapse 42 at any given time t. This strength determines the plasticity of neuron 40:

[0070] In this way, each neuron 40 can adjust its plasticity, enabling the network to easily and efficiently adapt to new data, including self-correction, mode switching, and adaptation to unexpected inputs.

[0071] Before execution, the network is initialized from the netlist according to the microarchitecture (because the microarchitecture defines the allowed network structure and behavior). Initialization involves initializing computational neurons 40 (including output neurons) and synapses 42. During initialization, at time... Set the activation value of each computational neuron 40 to 0 ( ), and each synapse 42 in the network has a fixed strength (Define the initial synaptic strength in the netlist).

[0072] In addition to initializing computational neurons 40 and synapses 42, the input and output neurons within the network are normalized to scale the inputs and outputs to suit the target control system. As a subset of computational neurons 40, the output neurons behave identically to computational neurons 40, except that their activation values ​​are collected to form the network's output for each clock cycle. However, the input neurons differ from both computational neurons 40 and output neurons. Input neurons cannot have input synapses, therefore their activation values ​​cannot be calculated using Equation 1 above. The activation values ​​of the input neurons are set by the user at the start of each clock cycle according to the target application. In practical applications, a network wrapper tailored to the target application is used to set the activation values ​​of the input neurons, which performs scaling, unit conversion, and any other requirements necessary to normalize the inputs and outputs of the neural network used as controller 20.

[0073] While the aforementioned properties are preferred options for the network, it should be noted that some network properties can vary. For example, alternative learning rules can be encoded. However, it is important that the learning rules are parameterized per synapse, which fixes certain graph structures and allows the optimizer to find the optimal learning rule for each synapse during network training. Similarly, alternative mathematical update rules for neuron activation can also be used. For example, alternative activation functions of Equation 1 can be used, and / or gain tuning can be incorporated. Neuron activation functions can include gain parameters to resist the tendency for the magnitude of activation values ​​in deeper neurons of the network to decrease.

[0074] Once the trained neural network is compiled into a netlist, the netlist and the runner together create the neural controller 20. The runner is a hardware or software implementation of the microarchitecture that enables the neural controller 20 to run on hardware. A software runner typically loads the netlist compiled from the trained and optimized network, while a hardware runner can be a manufacturing process that embodies the netlist in the target hardware (e.g., an IC, FPGA, or ASIC).

[0075] A variety of runners can be used, and the choice usually depends on the implementation. For example, a Python / Cython runner can be chosen for consumer CPUs, a C runner is suitable for embedded hardware, and a CUDA runner can be chosen for GPGPU implementations.

[0076] In some embodiments, a netlist may include a cryptographic signature. Signing the netlist serves as a security feature and prevents experimental networks from being deployed to production hardware. When a netlist is signed, the network operator verifies that a release state (specifying a completed development phase) is suitable for the application; this release state is encoded into the netlist as part of the cryptographic signature. Furthermore, in some embodiments, the netlist may be encrypted to prevent unauthorized use.

[0077] Once the netlist and the runner form the neural controller 20, the neural controller 20 can be deployed to control the target control process.

[0078] Luffy training method For reference Figure 3 In brief, creating a neural controller 20 first requires a trained neural network, which is then compiled into a netlist according to the microarchitecture. Figure 9 The flowchart illustrates a method 60 for developing and training a neural network for subsequent creation of a neural controller 20 according to an embodiment of the present invention, which will be discussed in more detail below.

[0079] like Figure 9 As shown, creating a trained network for subsequent compilation into a netlist involves multiple development phases. First, at step 62, the target control system is characterized and modeled. To characterize the system, relevant control system theory, physical dynamics, and sensor and actuator characteristics of the system need to be considered. System characterization can be achieved, for example, through direct experiments and data acquisition, communication with relevant personnel, and review of existing technologies and academic literature. Once the system is characterized, it can be used to develop a system model, and mathematical and software models can be generated. During model development, relevant phenomena (involving non-obvious physical phenomena occurring in the system) and dynamics can be captured with sufficient fidelity to create an accurate model. It should be noted that this model is only used during the training, validation, and verification of the network; in actual use, the controller 20 operates independently of this model. Therefore, even if an accurate model is required, this method does not incur high computational costs. MPC (an alternative control method to APC) also requires an accurate model; however, in MPC, the model runs in the control loop and is continuously updated, making MPC extremely inefficient.

[0080] Once the target control system is modeled (thus fully understanding the control problem), an AI curriculum is developed at step 64. The AI ​​curriculum defines a series of control challenges for the AI ​​network to perform within the system model. This curriculum includes multiple levels of progressively increasing complexity to guide the AI ​​network during training and expose it to all possible behavior spaces. The AI ​​curriculum also includes one or more reward functions that mathematically formulate the network's performance on a given AI curriculum.

[0081] Once the AI ​​curriculum is developed, at step 66, the candidate AI network is trained using the curriculum and an optimizer. A gradient-free optimizer, such as a neuroevolutionary optimizer (as described above), is used, and the optimization process continues until a predefined fitness is reached, or the optimization process is terminated by the AI ​​developer. The result of the AI ​​training is one or more candidate networks that may be effectively used as AI controllers. Subsequently, the candidate networks are validated against a validation curriculum that has similar statistical properties to the training curriculum.

[0082] Once training and validation are complete, AI validation is performed at step 68. Validation involves testing the candidate network with one or more validation challenges. Validation challenges are designed to test the network's extrapolation capabilities beyond its training set. For example, if a higher-fidelity simulation exists, the validation challenge can be performed in that higher-fidelity simulation. Training, validation, and validation ensure that the solution is stable and does not diverge rapidly when exposed to unseen conditions.

[0083] Once the candidate network is confirmed, hardware integration and testing are performed at step 70. Depending on the target control process, the neural controller 20 is integrated with the target hardware. The hardware is then tested, which involves designing a series of experiments in the real world and comparing the results with the same experiments performed in the system model to test for deviations between the model and reality. The performance of the neural controller is monitored in these controlled tests, and the results are then iteratively fed back to system modeling (step 62), curriculum development (step 64), and AI training (step 66). Once the neural controller 20 passes all the tests in the hardware, it is considered to have successfully bridged the reality gap and is suitable as a controller for the target control system.

[0084] Finally, at step 72, the neural controller 20 is deployed to the target control system. Key performance indicators (KPIs) or operational thresholds can be specified for the target control system and monitored during hardware deployment. For any controller, there are trade-offs between several KPIs. Examples of KPIs include time to reach the setpoint, overshoot, and energy reduction. More specific examples could include "the network must operate normally in winds up to 10 mph" or "the network must reach the setpoint within 10 seconds." Ideal controller behavior is context-dependent: some applications may need to reach the setpoint as quickly as possible regardless of overshoot, while others may need to minimize energy consumption at the expense of speed; therefore, the specific trade-offs between KPIs vary from application to application. Typically, additional constraints are defined for the motion of the target control system, such as position constraints or velocity "soft limits" (areas in the state space the controller should avoid) and "hard limits" (areas in the state space the controller must not access). KPIs are determined based on customer requirements during system characterization. Therefore, although KPIs are monitored during hardware deployment, they are fed back to model development and curriculum development and form part of AI validation and hardware testing.

[0085] Figure 10 It shows the relationship with Figure 9 A flowchart similar to the one shown. However, Figure 9 The flowchart is shown in sequence, where each step progresses sequentially to the next. Figure 10 A method 80 for developing and training a neural controller 20 is shown, which incorporates feedback 82 between different methodological steps. (See diagram.) Figure 10 As shown, each step can be fed back to any other stage, and in other embodiments (not shown), some method steps can be executed in parallel. An example of such an example has been described above, where hardware test results can be fed back to model development and curriculum development.

[0086] Figure 11A schematic block diagram of a system for creating a neural controller 20 according to an embodiment of the present invention is shown. A network (i.e., a population of candidate neural networks) is provided to a neuroevolutionary training process 90, which includes a gradient-free optimizer, such as an evolutionary algorithm 92, a curriculum 94 (containing a training curriculum and a derived validation curriculum), and a simulation 96 of a target control system. The neuroevolutionary system is initialized with one or more seed solutions, which may include pre-existing neural network structures. The evolutionary algorithm 92 trains and optimizes the network in the simulation 96 using the training curriculum, and then validates the network using the validation curriculum to generate candidate networks for the controller 20. The candidate networks are evaluated against the training and validation curricula 94, and a fitness value representing their accuracy is assigned to the candidate networks. If the candidate network is deemed to have passed training and validation (fitness value above a certain threshold), the candidate network undergoes validation 97 and is then output to hardware testing 98 (in a hardware test controller). If the candidate network passes all hardware tests 98, the optimized network is output from the system and deployed as the neural controller 20 of the target control system. If the candidate network fails the hardware test 98, a new network is provided and trained by the optimizer.

[0087] The following sections will discuss each step involved in creating and training the neural controller 20 in more detail.

[0088] Modeling For reference Figure 9 and Figure 10 The first step in developing the neural controller 20 involves creating a model or physical simulation 96 of the target control system. This physical simulation 96 provides an environment for training the AI ​​network and offers the ability to test failure scenarios that would be impractical or too costly to test in a real system.

[0089] To create a competent AI controller, a sufficiently accurate simulation of the target control system is required. To evaluate the quality of the simulation for AI training, three factors are assessed: the physical phenomena of the system (mechanics, control delays, sensor noise, etc.); the fidelity of these phenomena being modeled; and the degree of randomization of the applicable domain (e.g., variations in initial conditions, sensor parameters, etc.). These three factors should be sufficient to enable the neural network to be effectively transferred from the simulation to actual hardware.

[0090] However, in addition to creating a sufficiently accurate model 96, it is also desirable to speed up the time required to generate the network, thereby reducing the network training time. To find a balance between model accuracy and the time required to train the network in simulation 96, simulation development should focus on the main physical phenomena of the system rather than precise physical characteristics as much as possible.

[0091] In addition to considering the physical hardware and related physical characteristics of the target control system, the neural controller 20 should also be able to overcome any challenges associated with controlling the actual hardware. These challenges include defects in the sensing system, manufacturing errors and tolerances, and mechanical degradation. To account for these challenges, simulation 96 should include control parameters representing these challenges. All parameters included in simulation 96 can also be domain randomized according to the developer's determination.

[0092] Course Learning Following or concurrent with system modeling, AI Course 94 was developed to guide the evolution and training of AI networks. AI Course 94 includes a series of training levels. Each level contains a set of scenarios. These scenarios define different challenges with varying degrees of realism, typically corresponding to real-world situations in a target control system. As the AI ​​network evolves, it progresses through each level sequentially, expanding the network's capabilities. Training levels typically increase in complexity, with later levels including more complex challenges and confounding factors, enabling the AI ​​network to learn all possible scenarios in a real-world system. Often, a portion of the completed scenarios is retained (i.e., included across multiple levels of Course 94) to prevent the AI ​​network from forgetting previously learned scenarios while learning new ones (a challenge in training neural networks known as catastrophic forgetting). Another method to prevent catastrophic forgetting is to include scenarios of the same but not identical type in each level of Course 94.

[0093] In AI course 94, It is the set of all possible scenarios, usually an infinite set, while level 1 This is the level containing the simplest scenarios in AI Course 94. For example, Course 94 is used to develop controllers for training robot limbs. This can include the initial angle position. and setting value .but It comes from and The set of all possible combinations. From the set of all possible combinations, four scenarios can be selected as... The scenarios used. These scenarios can be selected using any appropriate method—such as grid search. Therefore, From Mid-sampling n A subset of the scenes, namely, .

[0094] AI Course 94 successfully trained an AI network to control a target system. Once developed, the training process embodied by Working Course 94, which is designed for a specific problem or control domain, can often be advantageously generalized to similar problems.

[0095] Network fitness and reward function To evaluate the accuracy of the trained AI network and, consequently, the success of the neural controller 20, one or more reward functions are defined and included in AI Course 94, where higher rewards correspond to better solutions. Reward shaping is a major component of the AI ​​training process because a good reward function encourages target behavior and discourages undesirable behavior. The network's behavior in each scenario is evaluated by the reward function (which may differ for each scenario). The solution to the reward function is assigned a fitness, where higher fitness corresponds to higher total rewards across all training scenarios.

[0096] The reward function can be calculated after each network execution or based on some summary statistics at the end of the scenario. The reward function can also be composed of other reward functions, such as in a weighted combination.

[0097] During training, the global optimizer tunes parameters to improve performance as measured by the provided reward function. Parameters are saved periodically during optimization. In the example of the global optimizer for evolutionary algorithm 92, the algorithm stores the network topology and parameters that provide the highest fitness; these parameters are called champion parameters. Whenever a new set of parameters emerges that improves fitness compared to the previous champion parameters, the training process stores this new set of parameters.

[0098] In some embodiments, the reward calculation includes a term that penalizes network complexity, particularly for network size and graph complexity.

[0099] Verification and Confirmation A common challenge in training neural networks is overfitting, where a network learns specific behaviors for its training scenarios, but these behaviors fail to generalize to the full problem domain from which those scenarios were sampled. Overfitted networks may perform poorly or fail entirely when exposed to new scenarios, even if these scenarios are identical in nature to those seen during training. To detect overfitting, all champion networks (networks created using champion parameters) pass a validation course, which is created by resampling the same set of scenarios used to define the training course.

[0100] In the example above, the training course includes levels. This level includes scenarios. Overfitting networks are problematic in specific scenarios. The individual performed well but not in the overall collection The network is performing poorly. The verification course will have levels. This level will include That is, for the same set of scenes New sampling. Validation levels may also sample a larger possible space, or be biased towards more difficult scenarios.

[0101] Validation exists to detect overfitting, and overfitting typically indicates that the optimization process has stalled. Early stopping can be performed using validation fitness, specifying that training should terminate if the validation fitness of N consecutive champion parameters is less than the observed maximum validation fitness.

[0102] Once a high-performance network or champion network completes its training course and performs well in validation, the champion undergoes confirmation. Confirmation involves testing the network on more extreme challenges, testing its extrapolation behavior and edge cases. The challenges used exceed the expected difficulty for controlling the actual hardware. Typically, these challenges are progressively increased in difficulty to determine the performance envelope of the champion's parameters. This validation set differs from the "test set" typically used to detect overfitting or undergeneralization during network training. Once confirmed, the champion network is tested in hardware, and if it passes all hardware tests 98%, the champion network is deployed to the target application.

[0103] Example Application – Controlling a Brushless DC Motor The neural controller 20 generated according to embodiments of the present invention can be used to control many different systems, one example being the control of a motor. A method for creating a neural controller 20 for controlling a motor is described in detail below, along with the results obtained when the neurally controlled motor is used to drive a robot limb.

[0104] In this application, the neural controller needs to drive a general-purpose current-controlled brushless DC (BLDC) motor and include closed-loop control of angular position and angular velocity, as well as additional configuration inputs for regulating behavior (such as safety constraints or context information).

[0105] System characterization and modeling The development of the neural controller 20 first involves modeling the target control system, namely the BLDC motor. It should be understood that any appropriate modeling technique can be used, but in this example, a physical model 96 based on first-principles electromechanical derivation is implemented. Although the target control system is a BLDC motor, a physical model 96 for a brushed DC motor is described here because, despite operational differences, a physical model 96 used to represent a brushed motor can also represent a BLDC motor.

[0106] In a brushed DC motor, a cylindrical rotating component known as the rotor has current-carrying wires wound along its axis of rotation, known as the armature. The rotor surrounds a stationary component called the stator, which generates a magnetic field (typically produced by opposite poles of permanent magnets) across the rotor and perpendicular to its axis of rotation. When the armature is energized, this magnetic field generates a Lorentz force on each armature conductor. Because the windings carry current in opposite directions on each side of the rotor axis, opposing forces are generated on opposite sides of the axis, and torque is produced on the rotor until the armature aligns with the magnetic field. To maintain torque, the direction of the current in the armature is switched by a rotary switch called a commutator according to an angle. Typically, the commutator consists of commutator segments arranged in a ring; this ring rotates with the rotor and establishes and disconnects connections with the stationary DC circuit via brushes.

[0107] In a brushed motor, for N armature windings, the commutator will have 2N commutator segments, which determines the granularity of current switching. For a two-commutator segment commutator, the torque on the rotor is... And to achieve constant torque The idealized torque produced in this way is:

[0108] in The magnitude of the armature current. This is the motor torque constant, which can be considered a constant value for a specific motor.

[0109] However, when a specific armature conductor rotates in a magnetic field, a Lorentz force is also generated, which is opposite to the direction of the current. This force is based on the rotor angular velocity. Reverse EMF :

[0110] This reverse EMF has the effect of reducing armature current:

[0111] in, and These represent the armature resistance and armature voltage (ignoring inductance), respectively. Since motors typically have a maximum operating voltage... Reverse EMF An upper limit is imposed on the armature current, which in turn imposes an upper limit on the rotor angular velocity.

[0112] In a BLDC motor, the electromechanical relationship between the rotor and stator is reversed: a typical BLDC motor uses permanent magnets in the rotor, which is controlled by electromagnets in the stator. Commutation is purely electrical, where the rotor position is synchronized with the alternating current flowing through each phase of the stator armature. The rotor's permanent magnets can be built-in, but are typically surface-mounted on the outside of a cylinder with alternating pole pairs at equal angular intervals. The stator electromagnets are solenoids wound around stator teeth, the gaps between which are known as slots. BLDC motors also include a DC-AC converter.

[0113] As briefly stated above, despite the significant changes in operating conditions, the idealized physical model of a brushed DC motor remains applicable to a BLDC motor. However, this idealized model is not accurate in real-world systems, where motors are subjected to a variety of forces, all of which must be considered when characterizing a BLDC motor. The forces considered in this control system will be discussed below.

[0114] Inertia, load, and gear transmission all affect the operation of the motor and are therefore included in the characterization of BLDC systems. Typically, for a motor to be usable, it must bear a load. The load acts in the same or opposite direction to the rotor's motion in two ways: by increasing the system's inertia J and by applying load torque. T R In a direct-drive motor (gearless motor), the load torque can be directly subtracted from the rotor torque. The motor's equation of motion in this case is:

[0115] It should also be understood that the rotor itself has inertia, therefore .

[0116] Some electric motors are equipped with gears, which are used to increase the motor's output torque. However, the trade-off is a reduction in output angular velocity. :

[0117]

[0118] Gears themselves also possess inertia; therefore, if measured separately at the input and output ends of the gears, the system's inertia will differ, with the rotor inertia scaled by the square of the gear ratio. Consequently, the equations of motion must be adjusted, and for a motor containing gears, the equations of motion become:

[0119] Ideal motors do not exhibit friction, while real motors do, and therefore friction must be considered. There are several classic models of motor friction; Table 1 below shows four of them.

[0120] Table 1. Types of friction in electric motors

[0121] Considering only viscous friction and Coulomb friction, the equation of motion for the direct-drive motor becomes:

[0122] And the equation of motion for the geared motor becomes:

[0123] in, and These represent the additional viscous friction and Coulomb friction introduced by the gears, respectively.

[0124] Cogging torque, caused by the attraction of the rotor permanent magnets to the stator teeth, can also affect motor operation. Therefore, in some embodiments, a cogging torque model may be included when characterizing the BLDC system. Various different models can be used, which will not be discussed in detail here. Similarly, in some embodiments, models representing any thermal effects or backlash of the motor may be incorporated.

[0125] Electronic controls, inductance, and heat generation can also affect motor operation. A typical motor accepts a target current. As input. In a BLDC motor, the input DC voltage is converted to three-phase AC voltage, where the voltage through each phase is modulated by pulse width modulation (PWM). Inductors smooth the input voltage, allowing each phase to operate at different target currents, which are typically coordinated by integrated electronics with the rotor position to eliminate torque ripple. The effective current relative to the rotor torque. Equation 7 provides the definition. Equations 8 to 12 above define the characteristics of the BLDC motor and are used to develop model 96.

[0126] It should be noted that while various secondary effects, such as cogging and friction, can lead to more accurate simulations, these secondary effects can be removed from the formulas by selecting appropriate model parameters. The four non-zero parameters required to specify the motor are only... , , and It should also be understood that, although not included in this example, other secondary effects can be considered when developing model 96 for a BLDC motor. For example, it is known that both the mechanical and electrical aspects of a motor generate heat, so temperature can also be considered, as it affects the resistance of the electrical system, thus affecting the torque output. In some BLDC control systems, backlash can also be considered and incorporated into system model 96. In systems that transmit mechanical forces, such as gear trains, mechanical connections are never perfect. The clearance between gear teeth results in a "dead zone," in which no force is generated between the gear teeth, allowing the first gear to move freely while the second gear transmits no force. This is known as backlash.

[0127] Using the characteristics defined above (Equations 8 to 13, and if included, the cogging torque model, any thermal effects, or backlash), a dynamic motor simulation 96 for a BLDC motor can be developed, wherein the simulation 96 is defined using a series of equations.

[0128] time t instantaneous angular acceleration With discrete time step The propagation is given by the following formula:

[0129] This includes the torque generated by the motor (including all influencing factors such as friction and cogging effect), and the total inertia of the motor, including gears.

[0130] The control variable is the target output current of the motor. Thus, the rotor motion generates and instantaneous reverse EMF Calculate the actual armature current :

[0131] Therefore, the state of the system at time t can be determined by the angular position. angular velocity Load torque and load inertia Provided.

[0132] It should be noted that the state observed by controller 20 has a delay of up to one time step and is subject to offset by noise, and this delay can be incorporated into model 96.

[0133] Used to update angular velocity and angular position The equation is given by an independent Welley integrator, which is determined as follows:

[0134]

[0135] in This indicates the variable being considered, therefore the above formula also applies to angular velocity. and angular position Both. Since the above equation is integrated with respect to the second derivative, The value depends on the jerk. It can be simply calculated using backward differencing: .

[0136] The above formulas (Equations 14 to 17) define the simulation dynamics and the environment for training the neural network. During model development, the first model iteration may not be accurate enough because it may overlook many minor physical effects. However, hardware testing can be performed in parallel with system characterization and modeling (e.g., ...). Figure 10 (As shown) Any deviations between the physical behavior and model 96, such as high-frequency response (a consequence of unmodeled dynamics), are illustrated. Model 96 can then be enhanced and refined based on feedback 82 from hardware test 98 to generate a more accurate model 96.

[0137] The controller 20 is configured to control the motors to drive the robot's limbs. Therefore, the controller 20 sets an appropriate target current. To drive the motor to reach the set value Its goal is to reach the setpoint in a critically damped manner, that is, to reach the setpoint in the shortest possible time while maintaining a steady-state error, no overshoot, and no oscillation based on the observed motor conditions. The controller 20 is designed to drive the motor to a defined setpoint under various load torques and inertia, including loads that vary with time or location (such as gravitational torque).

[0138] Controller 20 is also designed to constrain the reachable position of the motor between two hard limits, which act as physical barriers: when the simulation exceeds the hard limit value, the position is clamped at the hard limit, the velocity is set to zero, and the integrator state is reset (Equations 16 and 17, corresponding to the values ​​at the previous moment). Controller 20 also restricts motion between two configurable boundaries, known as soft limits; in this example, these two soft limits are two angular positions. and .when When the soft limit is exceeded, controller 20 will stabilize at the soft limit position and will not exceed it. During training, neural controller 20 thereby learns a policy that maps state measurements to control variables such that:

[0139] Course Design AI Course 94 is designed for training neural networks and can be performed after or in parallel with system modeling. The first level of AI Course 94 defines the simplest scenario for the target motor, capturing the most basic version of the problem. The simplest problem is achieving a given setpoint without external forces, i.e., no motor load, no changes in model parameters (e.g., no change in inertia, constant torque, etc.), no sensor noise, no friction, and no cogging. In this simplest level of AI Course 94, the reward function used to evaluate the network's accuracy does not consider soft or hard limits, and the setpoint is defined as a constant.

[0140] In later levels of the course, the aforementioned factors such as load, sensor noise, friction, tooth grooves, and model parameter variations are gradually introduced, along with randomly generated setpoint waveforms that can change over time.

[0141] The following is a detailed description of course 94 specifically for BLDC motor development. First, domain randomization is employed to minimize overfitting during neuroevolutionary optimization. The domain variables to be randomized are the motor parameters. , , , and waveform , , and constants , , , (Sensor noise) and (Delay amplitude) (These variables are sampled from a uniform distribution within an appropriate range of values). In randomized soft-limiting... and When using soft limits, it is appropriate to use the distance between them rather than their absolute positions.

[0142] AI Course 94 is designed to cover a wide range of representative control tasks and therefore uses two types of waveforms to define variables: step waveforms (where the value remains constant between randomly selected points and then changes to the next value); and arbitrary waveforms that contain both smooth and discontinuous changes over time. The waveforms are constructed from piecewise combinations of easing functions 101 (such as...). Figure 12 As shown), the easing function is a commonly used function in animation interpolators. Waveform example 103 can be seen as follows: Figure 13 As shown. , , One or all of these waveforms can be used, or they can remain constant throughout the scene.

[0143] AI Training In Lesson 94, the network's performance is evaluated for each scenario. Evolutionary Algorithm 92 is used as the neuroevolutionary optimizer, specifically the maximization optimizer. Therefore, performance evaluation is represented in the form of rewards. However, in this particular application related to motor control, the network's performance is evaluated using a loss function, as this is more intuitive, and each loss can subsequently be converted into a reward. The primary objective during optimization is to bring the network to a set waveform. Therefore, the loss function penalizes deviations from a set value. An example of a loss function could be:

[0144] in For setting value, The allowable setpoint error. Any appropriate loss function can be used, but it should be a composite reward function that includes the distance, velocity, and jerk magnitude from the setpoint.

[0145] The total reward for a scenario is simply equal to the average reward obtained per time step. A scenario is considered complete when this value is greater than a threshold selected from [0, 1] through iterative experiments.

[0146] In this example, Course 94 comprises several levels, with each level introducing a new level of challenge. Level 1 uses motor parameters shared across all scenarios ( , , , , The parameters include load inertia and soft limits, frictionless operation, control delay, and sensor noise. Variations in these parameters are gradually introduced in subsequent levels to create more challenging scenarios for network training. Within each scenario at each course level, each parameter can be the same, a constant, or a time-varying value, and for some levels, the sampling method remains unchanged from the previous level.

[0147] Each level of course 94 provides controller 20 with a balanced combination of set waveforms for matching: stillness ,fixed The loss function is manually adjusted to penalize deviations from the setpoint most on stationary waveforms, followed by fixed waveforms, and then step and arbitrary waveforms. The fundamental reason for including this mix is ​​to prevent catastrophic forgetting, i.e., networks recently trained to follow step functions may struggle to follow smooth waveforms, or networks capable of handling changing waveforms may lose their ability to remain completely still when appropriate.

[0148] Subsequent course levels also include some scenarios from the earlier levels to combat catastrophic forgetting in progressively more difficult problem domains. Based on the characteristic speed of the motor used in this example, the training scenario lasts 30 seconds, and the simulation runs at a frequency of 500 Hz. For the validation course, the scenario duration is extended to 150 seconds to ensure malleable stability.

[0149] The neural network was trained using an evolutionary algorithm 92 that followed the above-described curriculum. This algorithm 92 was distributed across a computer cluster using 320 CPUs. Algorithm 92 employed ranked parent selection, meaning that better individuals were prioritized for mutation, and a survivor selection strategy combining steady-state selection and tournament selection, while also employing an elitist approach, meaning that some individuals from previous generations were retained, including those with the highest fitness. The network was evaluated against the training curriculum, followed by a validation curriculum. Once the candidate network was trained, the network that performed best on the validation curriculum (determined by fitness) was then tested against the validation set. Validation was performed by extending the range of system latency, load torque, and inertia beyond what was expected in the real world. Once the candidate network passed validation, it was tested in simulation experiments and on actual hardware to demonstrate its ability to migrate from simulation to reality (i.e., its ability to effectively migrate from simulation 96 to hardware).

[0150] Hardware testing After training and validation, hardware integration and testing are performed. Hardware testing is an integral step in creating the trained neural network for the neural controller 20, as the results are fed back into model development to create an accurate model 96, and also into curriculum development. In this example, an ODrive 270kv motor from ODrive Robotics (known as a test motor) is used for experimentation. Subsequently, the network is retrained using two additional motors, RMD X8-Pro and X10, for further experimentation. The specifications of the three motors are listed in Table 2.

[0151] Table 2. Motor Specifications Used in Hardware Testing Experiments

[0152] The trained and validated neural network must be integrated into the hardware for testing, and the integration and testing method depends on the specific BLDC motor. For example, for the test motor, the ODrive control board connects to a consumer PC via USB, while the RMD X8-Pro and X10 motors use a custom protocol for input / output with the motor's onboard electronics on the Controller Area Network (CAN) bus. To integrate the network into both types of controllers, a script was written that loads the netlist (generated from the trained neural network and microarchitecture) and executes the resulting network using a Python runner. Considering the test motor first, to test the trained network, the ODrive can be mounted on a test bench with a metal rod attached to demonstrate angles. Counterweights can be added to the rod to change the system's mass and inertia.

[0153] The verified and validated network was first tested on a test motor (ODrive) on a test bench. This test bench has an arm of length l connected to the gear output side, capable of supporting an adjustable mass m. This allows for variations in load torque and inertia between different scenarios. The motor shaft is perpendicular to the direction of gravity. Therefore, the load torque... The load inertia was set to zero at the start of the experiment. It remains constant throughout the scenario. Controller 20 is tested for two step responses: from vertically downward to vertically upward, and from vertically downward to horizontally to the right (i.e., half-turn and quarter-turn step responses), while varying the load mass. This experimental setup is known as the pendulum experiment.

[0154] Once Course 94 could reliably generate a neural network that performed well on the test motor, namely the second type of motor, the X10 motor, switching to the X10 motor meant that Course 94 had to be retuned. The slower dynamic characteristics of the X10 motor allowed for a reduction in the network's operating frequency, thereby improving the computational efficiency of the optimization process. After retuning, the AI ​​course generated a neural network that successfully controlled the X10 motor. The new AI network (tuned using the X10 motor) was also able to control the X8 motor, a similar motor. However, it is worth noting that, given sufficient external load, the generated network could also control the ODrive 270 motor (the test motor, which is far beyond the training domain of the X10 motor). Without load, the network generated by the X10-adapted AI course 94 produced high-frequency oscillations when controlling the ODrive motor, but was otherwise acceptable. Therefore, the developed neural controller 20 can control a wide range of BLDC motors.

[0155] Figure 14The experimental setup for the pendulum experiment is shown in the figure, where the BLDC motor 108 is configured to move the robotic arm 110 from a vertically downward position to a horizontal position or a vertically upward position. A counterweight 112 can be attached to the end of the robotic arm 110. Figure 15 The diagram illustrates the ability of the neural controller 20 to control the BLDC motor 108 to drive the robotic arm 110 in six different motion scenarios, each with different setpoints and / or different external forces. For each diagram shown, the letters correspond to the desired setpoint position (V = vertically upward, H = horizontal), and the numbers correspond to the number of counterweights 112 attached to the robotic arm 110 (1, 3, or 6). In each diagram, the target bandwidth of the setpoint is the setpoint + / - 5 amplitudes. Figure 15 The results show that in all six test motion scenarios, the neural controller 20 reaches the set value at the fastest speed allowed by the system, and reaches and maintains that set value within the defined target range. It should be noted that the small ripples and slight undershoots observed in the six-board test are due to human error caused by premature termination of network training; further training and tuning of the fitness function will eliminate this behavior.

[0156] In contrast. Figure 16 An example of PID control is shown, illustrating how a PID controller controls the same motor 108 to drive a robotic arm 110 in a manner such as... Figure 15 The capabilities are shown for the same six motion scenarios. The PID controller is tuned separately for each of the six motion scenarios (PID tuning can be done, for example, manually or using the Ziegler-Nichols method as in this example, both of which require active involvement from the control engineer), resulting in six individually tuned PID controllers. Figure 16 In the graph, each row corresponds to a individually tuned PID controller, and each column corresponds to a test motion scenario. First, the graphs marked with an asterisk highlight cases where the PID tuning is perfectly matched to the motion scenario. In these cases, the PID controller accurately controls the robotic arm 110 and rotates it to the desired setpoint. Each properly tuned PID exhibits a fast rise time with minimal overshoot or deviation. However, Figure 16 The graph without an asterisk shows the motion results obtained when a PID controller is designed to control an untuned motion scenario (i.e., the PID is mismatched with the scenario). The results indicate that a PID controller tuned for a specific motion will undershoot or overshoot the desired setpoint when used to control different motions. For example, when a V6 (vertical, 6-piece) PID is applied to a V1 (vertical, 1-piece) scenario, the speed at which the controller approaches the setpoint is significantly reduced because the controller's dynamic characteristics are tuned for a heavier weight than the weight to which the PID is applied.

[0157] The inability to control untuned motion types is a major limitation of PID controllers. Untuned PID controllers are known to operate outside the linear region of the PID controller. The linear region is where PID control is accurate; however, this region is very small. Therefore, PID controllers may only be applicable to a very narrow domain. Figure 15 and Figure 16 The results presented show that the neural controller 20 of the present invention has a wider operating range than a tuned PID.

[0158] Two additional experiments were conducted to evaluate the neural controller's ability to control the BLDC motor 108 in an untrained system. In the two experiments discussed below, the PID controller could not be tuned for comparison because the time cost of tuning a gain-scheduled controller to match such dynamic conditions was too high.

[0159] In these experiments, the target control system is a multi-jointed robotic limb 120, modeled after the torque and geometry of a human leg (a 3-DOF hip joint 124 includes three motors, and a 1-DOF knee joint 122 includes one motor). Each motor instantiates a controller 20 (i.e., a multi-network configuration), and each agent is able to operate well under dynamic loads of increased mass, external forces, and torque, and is able to counteract reflected torques from other agents. In simple scenarios where PID is applicable, an agent can be qualitatively equivalent to a perfectly tuned PID, as described below.

[0160] In the first follow-up test, the two motors 122 and 124 were mechanically coupled in a configuration simulating human leg 120, an arrangement not explicitly shown during network training. The hardware setup for this experiment is as follows: Figure 17 As shown, the "hip" motor 124 and the "knee" motor 122 (both RMD x10) are driven simultaneously. Each motor 122, 124 is controlled by an independent instance of neural controller 20, so each controller 20 must cancel the action of the other controller to achieve its own setpoint. Controller 20 is fed a waveform consisting of piecewise step functions, which are very similar to the step waveforms seen during training, with some steps occurring simultaneously and others occurring randomly. The difficulty of this challenge is further exacerbated by an unplanned mechanical failure in the hip joint gear train prior to the experiment, which introduced extreme backlash at the hip joint. Prior to this experiment, the network was retrained for the RMD motors, with a performance envelope covering both motors.

[0161] The results of the experiment are as follows: Figure 18As shown. Figures A and B show the joint position (angle) 130 measured by a motor encoder located between the output side of the motor's internal gear train and the externally applied gear train; a setpoint waveform 132 with a tolerance of 1.5° is also shown. Figures C and D show the output current 142 of the controller 20, superimposed with another joint position waveform 144 to highlight the current variation used to counteract the reflected torque. Figure 18 Figures C and D also show the current limit 140. The effects of mechanical failure on the hip joint are very evident in the violent swings in the current output 142 and angular position 130 (Figures C and A, respectively) over, for example, 14–19 seconds. Nevertheless, from the encoder's perspective, the hip joint motor still strives to track the setpoint (the actual angle of joint 124 on the gear train output side does not change directly with the encoder angle due to extreme slippage in the gears and cannot be measured); the downstream knee joint 122 controller is able to adjust the reflected torque from the upstream joint 124 and the high-frequency oscillations in its own relative angular position. This can be seen from the distance between angle 130 and setpoint 132, which never exceeds the 1.5° tolerance after stabilization (Figure B).

[0162] Each controller 20 exhibits clear signs of response to the movement of the other controller, such as the output of the hip joint at approximately 32 seconds (Fig. C), and the output variation of the knee joint between approximately 12.5 and 32 seconds (Fig. D), while maintaining a nearly completely stationary position. It should also be noted that the balancing current output of the knee joint 122 changes with the angle of the hip joint 124, for example, the output current differs between 20 and 30 seconds in Fig. D. Therefore, the neural controller 20 is able to control motors 122 and 124 in this experiment, and the system is its untrained system.

[0163] In the third test scenario, the upstream motor 124 (hip joint motor) was deactivated, allowing the downstream motor 122 to swing freely. Therefore, this configuration is similar to the motorized joint in the middle of a freely swinging pendulum, such as... Figure 19 As shown. Each time the joint itself applies torque, the physical orientation, and consequently the direction of gravity and the reflected torque, oscillate. In this experiment, an additional mass is added. m Attached to the end of a robot limb, and placed in... Changes occur within the range. A ladder waveform is used as the test waveform, from... The full extension to The complete contraction changes the direction of relative gravity with each step.

[0164] The result of this configuration can be found in Figure 20As shown in the figure. It should be noted that Figures A through D all contain traces related to the aforementioned experimental mass options (i.e., knee joints with no weight 154, 1kg weight 156, 2kg weight 158, and 3kg weight 160, respectively). Further note in Figures A, B, and C that the four traces largely overlap. Figure A shows the angular position of knee joint 122. θ Although it contains four lines, the angular positions in all experiments θ Similar. Figure B then shows the positional error for each experiment. θ t - θ | and settings Figure 152 and the tolerance region (shaded area), and Figure C show the positional deviation for each experiment. In Figures B and C, the graphs for each experiment are similar, except that the 3kg experiment 160 exhibits overshoot beyond the tolerance region, and because the controller 20 was unable to match the setpoint at each angle (at t>110 s) due to the manually applied current limit 162 of 5 A. 152. The current limit 162 is shown in Figure D, which also shows the target currents for four experiments 154, 156, 158, and 160.

[0165] exist At that time, the knee joint is fully extended; At this time, knee joint 122 is fully bent. As knee joint 122 bends, the system's center of mass shifts, causing the free hip joint 124 to rotate, ensuring that the center of mass remains directly below hip joint 124. This ensures the direction of gravity. Follow Smooth changes. Furthermore, when controller 20 moves, this causes the system to oscillate, resulting in a slight change in the direction of gravity, which controller 20 must adjust to. With Change, change The influence of stable signed position error Clearly visible in the image. When considering the external gear system, the training error tolerance is... The controller 20 remains stable within the tolerance range (allowing backlash), with no significant overshoot and a very fast rise time.

[0166] It is worth noting that rise time, settling time, and overshoot (i.e.) θ ( t The waveform remains consistent under different loads, therefore consistent with the average position. The deviation remained less than 0.5° throughout the experiment. Controller 20 requested different target currents for each load and setpoint. I t This indicates that it is due to the active suppression of external torque, rather than a random result of the passive controller.

[0167] The results show that the neural controller 20 developed for BLDC motor control significantly outperforms PID control in several challenging control tasks. Firstly, the neural controller 20 has a wider operating envelope than PID, such as... Figure 15 and Figure 16 As shown, the neural controller 20 demonstrates competitive advantages over specially tuned PID controllers in various scenarios. Furthermore, subsequent experiments demonstrate its effectiveness in coupled motors (…). Figure 18 ) and free-swinging pendulum ( Figure 20 The results in the paper demonstrate the ability of the neural controller to precisely drive robot limbs in scenarios where PID is difficult or even impossible to tune.

[0168] In the example applications described above, the AI ​​controller 20 of the present invention is used to control motorized joints, and it has the potential to be deployed in several applications in engineering, robotics, mechatronics, and manufacturing, as well as many other industries that require the automation of motorized joints. Some examples include controlling drives for devices such as: robotic arms in factories; joints of walking robots; cranes carrying inertial loads in docks; ball mills; conveyor belts; automatic doors or pistons; and grinders for ore processing, but many other applications exist. In particular, due to the high operating frequency of the network, the AI ​​controller 20 of the present invention is suitable for applications that require rapid responses to unpredictable external forces or environmental changes. For example, a typical control loop for an unmanned aerial vehicle (UAV) operates at 400 Hz; a typical deep network may execute at speeds below 50 Hz on available hardware, which is too slow for controlling an aircraft. Therefore, faster networks can be used for a wider range of applications.

[0169] In the presence of external forces, the neural controller 20 of this invention outperforms PID and other control techniques in terms of energy efficiency by adapting to the force when it is directed toward the controller setpoint. Furthermore, the AI ​​training process can generate networks in different operating domains using the same or slightly tuned curriculum through simple reconfiguration (modifying baseline model parameters to match the target motor and reducing the network frequency). Overall, this demonstrates the ability of the AI ​​controller 20 of this invention to model second-order (or lower-order) dynamic systems at high frequencies, and its ability to outperform any single PID controller in second-order (or higher-order) systems. Since PID control is also a widely used control technique, the AI ​​controller 20 of this invention can replace a single tuned PID controller wherever a single tuned PID is used, providing a more efficient and accurate control method.

Claims

1. A method for generating a dynamic system controller for controlling a dynamic system, the dynamic system controller comprising a neural network, the method comprising: The dynamic system is modeled to generate a system model; Define a control challenge training set for evaluating candidate neural networks, wherein the control challenge is derived from the system model; An optimized neural network is generated from a candidate neural network population using a neuroevolutionary optimizer, wherein the candidate neural network population is evaluated using the control challenge training set; The optimized neural network is validated using a control challenge validation set derived from the system model; The optimized neural network is deployed into a hardware test controller for the system; Evaluate the performance of the hardware test controller in hardware testing; In addition, if the hardware test controller fails the hardware test, a further optimized neural network is generated and the verification, deployment and evaluation steps are repeated; Alternatively, if the hardware test controller passes the hardware test, the optimized neural network can be deployed in a dynamic system controller for controlling the dynamic system.

2. The method for generating a dynamic system controller for controlling a dynamic system according to claim 1, wherein, The candidate neural network group is a seed network with a predefined neural structure.

3. The method for generating a dynamic system controller for controlling a dynamic system according to claim 1 or 2, comprising: Generating a further optimized neural network includes updating the system model and / or defining a further set of control challenges.

4. The method for generating a dynamic system controller for controlling a dynamic system according to any one of the preceding claims, wherein, The neural network includes neurons and synapses configured to form a directed graph structure, wherein at least some of the synapses in the neural network are configured to have strengths that can change during network operation.

5. The method for generating a dynamic system controller for controlling a dynamic system according to any one of the preceding claims, wherein, The control challenge training set includes a series of control challenges with progressively increasing complexity.

6. The method for generating a dynamic system controller for controlling a dynamic system according to any one of the preceding claims, wherein, The control challenge validation set includes control challenges that exceed the expected difficulty of controlling the dynamic system.

7. The method for generating a dynamic system controller for controlling a dynamic system according to any one of the preceding claims, wherein, The dynamic system controller is configured to receive the current system variables of the dynamic system and control the dynamic system toward a set value by outputting control signals.

8. The method for generating a dynamic system controller for controlling a dynamic system according to claim 7, wherein, The dynamic system includes an actuator, and the dynamic system controller is configured to output the control signal to drive the actuator toward the set value.

9. A method for controlling a dynamic system using a dynamic system controller according to any one of claims 1 to 8.

10. A dynamic system controller according to any one of claims 1 to 8 for use in controlling a dynamic system.

11. A dynamic system controller generated by the method according to any one of claims 1 to 8.

12. The dynamic system controller for controlling a dynamic system according to claim 11, comprising an input terminal for receiving set values ​​of system parameters and current system values, and an output terminal for outputting a control signal to control the dynamic system toward the set values, wherein, The optimized neural network is configured to generate the control signal based on the received setpoint and the current system value.

13. A neural network for a dynamic system controller of a dynamic system, the neural network comprising: Multiple neurons n and synapse e The neural network is configured to form a directed graph structure, wherein at least some of the synapses in the neural network are configured to have synaptic strengths that can be changed during network operation.

14. The neural network according to claim 13, wherein, Each neuron n When executing a clock instance, an activation value is generated based on the input activation value calculated in the previous clock instance execution.

15. The neural network according to claim 13 or 14, wherein, The synaptic strength of at least some synapses is configured to change based on previous neuronal activation values ​​during network operation.

16. The neural network according to any one of claims 13 to 15, wherein, The synapse e It includes a set of processor synapses and a set of regulator synapses, each processor synapse being configured to contribute to the activation of the neuron, and each regulator synapse being configured to contribute to the instantaneous learning rate of the neuron.

17. The neural network according to claim 16, wherein, The neuron includes an activation function and an instantaneous learning rate, and the synapse includes a learning rule encoded as a function of the current synapse strength, presynaptic activation value, postsynaptic activation value, and the learning rate. The activation function, instantaneous learning rate, and learning rule are used to calculate the strength of each synapse.

18. The neural network according to any one of claims 13 to 17, wherein, The network encodes multi-directed graph structures.

19. The neural network according to any one of claims 13 to 18, wherein, The initial synaptic strength is configured to be quantized.

20. The neural network according to any one of claims 13 to 19, wherein, Each neuron includes an activation function containing gain parameters.

21. A dynamic system controller for controlling a dynamic system, comprising the neural network according to claims 13 to 20.

22. The dynamic system controller according to claim 21, comprising an input terminal for receiving the current system value of the dynamic system and an output terminal for outputting a control signal to control the dynamic system, wherein, The neural network is configured to receive the current system value as its input, and the controller is configured to determine the control signal based on the output of the neural network.

23. A controller for controlling a dynamic system, comprising: It is used to receive the set values ​​of the control variables of the dynamic system and the input terminal of the current system variables of the dynamic system; Processing device for calculating control signals to control the dynamic system toward the set value; The output terminal used to output the control signal; The processing device includes a trained neural network having neurons and synapses connected in a directed graph structure, wherein at least some of the synapses in the neural network are configured to have synaptic strengths that can be changed during network operation.

Images