Controller with neural network and improved stability

Abstract

To provide controllers for generating control signals of computer controlled devices.SOLUTION: A neural network configured to map a sensor signal to a raw control signal can be applied to a current sensor signal. A projective function can be applied to the raw control signal to obtain a stable control signal for controlling a computer controlled device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Field The subject matter of this disclosure relates to a controller for generating control signals for a computer-controlled device, a control method for generating control signals for a computer-controlled device, a training method for training a neural network for use in controlling a computer-controlled device, and a training system for training a neural network for use in controlling a computer-controlled device. [Background technology]

[0002] background Despite recent impressive advances and state-of-the-art performance in many tasks, deep reinforcement learning has proven limited in its application in "safety-critical" domains, where erroneous actions during training or implementation can have substantial effects on the controlled system. In contrast, the field of robust control dates back decades and is capable of providing strict bounds on when a controller will succeed or fail to control a system of interest. Robust control techniques offer provable guarantees that the resulting system will be stable, particularly if the controlled system can be appropriately constrained in a specific manner. However, the simple nature of the measures resulting from certifiable robust control techniques, e.g., often linear, often limits performance in typical scenarios.

[0003] The design of feedback controllers for dynamical systems with uncertainties and / or unknown disturbances is addressed in robust control. Specifically, robust control approaches aim to design controllers with guaranteed performance even for worst-case realizations of uncertainties or disturbances in the system. In H∞ control, the goal is to stabilize the system while attenuating the effect of exogenous disturbances on some performance output (e.g., LQR cost). This effect is characterized by the L2 gain of the disturbance-to-output map, defined as the ratio between the L2 norm of the output and the disturbance. Despite providing performance guarantees, robust control approaches are usually overly conservative due to considering worst-case scenarios or being limited to a linear class of controllers.

[0004] Many classes of robust control problems, even those originally formulated in the frequency domain (e.g., an alternative approach to characterizing dynamical systems), can ultimately be formulated using linear matrix inequalities (see, e.g., "Linear Matrix Inequalities in System and Control Theory" by Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan). Thus, the resulting control problems can often be formulated as semidefinite programs that can be solved to create controllers for modest-sized domains using off-the-shelf software.

[0005] WO 93 / 00618 discloses that an adaptive control system uses a neural network to provide adaptive control when the plant is operating within its normal operating range, but transitions to another type of control when the plant's operating conditions move outside the normal operating range.

[0006] Reinforcement learning, particularly deep reinforcement learning (DRL), in which optimal control strategies are approximated by neural networks, has shown impressive results in learning a variety of complex control tasks. However, due to its lack of safety guarantees, DRL has primarily been applied to simulated environments or highly controlled real-world problems where potential system failures are unlikely or not costly. For DRL methods to be employed in safety-critical settings, it is necessary to combine these DRL methods with some form of safety guarantee, as addressed in the present invention. In practice, it has been found that control strategies approximated by neural networks can completely fail when faced with, for example, sensor readings or disturbances that are slightly different from those encountered during training. In particular, it has been found that under adversarial disturbances, the stability of the control strategy can be completely lost, even if the magnitude of the adversarial disturbance is limited. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. 93 / 00618 [Non-patent literature]

[0008] [Non-Patent Document 1] “Linear Matrix Inequalities in System and Control Theory” by Stephen Boyd, Laurent El Ghaoui, Eric Feron and Venkataramanan Balakrishnan Summary of the Invention [Problem to be solved by the invention]

[0009] What is desired is a safe reinforcement learning method that can implement control measures while maintaining system stability. Furthermore, it is desirable to use a characterization of global uncertainty, as in, for example, robust control, rather than adopting local smoothness assumptions on the dynamics or costs. [Means for solving the problem]

[0010] overview It would be advantageous to have an improved controller for generating control signals for computer-controlled devices. For example, a device may have multiple, complex, interacting parts. For example, these multiple parts can be viewed as or approximated by a single dynamical system.

[0011] The controller can apply a neural network to the sensor signals. The neural network is trained to control the device in some manner, for example, using direct search or reinforcement learning, so that the device reaches a specific state or follows a trajectory. The control can also be limited to a subset of states, such as the position of a vehicle, e.g., a car or drone, thereby controlling the car to stay within its lane or a drone to stay on a two-dimensional plane. However, rather than directly using the output of the neural network, e.g., the raw control signal, to control the device, a projection function is applied to the raw control signal to obtain a stable control signal. The projection function maps the space of raw control measures to a space of predefined stable control measures. As a result, neural network-based control is limited to control measures that are predefined to be stable. For example, the control measures can be predefined to be stable, e.g., by requiring the control measures to satisfy a stability equation, e.g., by Lyapunov reduction. The stable control signal can be used to control a computer-controlled device. A neural network can be trained with a projection function to select a good control strategy under the constraint that only stable control strategies can be used.

[0012] In one embodiment, the projection function is a piecewise continuously differentiable function. A piecewise continuously differentiable function has the advantage that, for example, reinforcement learning can be implemented. Note that the projection function can be a piecewise continuously differentiable function even when implemented as an optimization layer. Reinforcement learning can adjust the neural network to reduce a loss function. For example, the loss function can measure how close a system including the neural network is to reaching a goal, such as reaching a predetermined state, following a trajectory, maintaining constraints on all or some state variables, etc.

[0013] For example, a neural network can be tuned to decrease the loss function over multiple training steps. The training steps can be repeated, for example, until a desired accuracy is reached, a predefined number of iterations, until the loss function no longer decreases, etc. Neural networks can be tuned using a number of training algorithms, including backpropagation.

[0014] An interesting advantage of using neural networks in the front end is that it allows for a wide variety of sensor inputs to be used. Neural networks can be provided with processed information, such as the position or velocity of one or more pieces of equipment, but can instead learn to derive relevant information themselves. For example, a sensor system can include one or more of an image sensor, radar, lidar, pressure sensor, etc. Thus, sensor signals can include two-dimensional data, such as image data. For example, in the case of a robotic arm controller, sensor data can include one or more images depicting the robotic arm.

[0015] The controller can be used in a wide variety of control tasks. For example, the device can be one of a robot, a vehicle such as an automobile, a household appliance, a power tool, a manufacturing device, a drone such as a quadrotor, etc. In one embodiment, the controller not only provides robust control guarantees but also leverages the power of deep reinforcement learning. It can respect stability specifications generated by traditional robust control methods under different models of system uncertainty, while at the same time obtaining improved control.

[0016] For example, robust control methods such as H∞ control generate both stabilizing controllers and Lyapunov functions that demonstrate the stability of the system under certain worst-case perturbations. These specifications can be used to construct a new class of reinforcement learning tools that project nominal, e.g., raw, controls onto a space of stabilizing controllers specified by the robust Lyapunov functions. The raw controls can be nonlinear and based on neural networks. The result is a nonlinear control tool that can be trained using deep reinforcement learning, yet is guaranteed to be stable under conditions comparable to simple, conventional robust controls.

[0017] The present embodiment has been shown to improve upon conventional LQR while remaining stable even to worst-case tolerable perturbations of the underlying dynamical system, unlike non-robust LQR and neural network methods.

[0018] The controller and training system may be electronic devices and may include a computer. The control methods described herein are applicable to a wide range of practical applications, including the control of computer-controlled devices such as fully or partially automated vehicles, robotic arms, and drones.

[0019] A further aspect is a training system configured to train a neural network used in the controller. Further aspects are a control method and a training method. An embodiment of the method can be implemented on a computer as a computer-implemented method, or by dedicated hardware, or by a combination of both. Executable code for an embodiment of the method can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product includes non-transitory program code stored on a computer-readable medium for performing an embodiment of the method when the program product is executed on a computer.

[0020] In one embodiment, the computer program comprises computer program code adapted to perform all or part of the steps of the method embodiments when the computer program is run on a computer. Preferably, the computer program is embodied on a computer readable medium.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS Further details, aspects and embodiments are described, by way of example only, with reference to the drawings, in which elements are illustrated for simplicity and clarity and are not necessarily drawn to scale, and in which elements corresponding to elements already described may have the same reference numerals. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram of an example of an embodiment of a controller for generating control signals for a computer-controlled device. [Figure 2a] 1 is a schematic diagram of an example of an embodiment of a physical system; [Figure 2b] FIG. 1 is a schematic diagram of an example of an embodiment of a robotic arm. [Figure 3] FIG. 1 is a schematic diagram of an example of an embodiment of a controller for generating control signals for a computer-controlled device. [Figure 4] FIG. 1 is a schematic diagram of an example of an embodiment of a training system for training a neural network for use in controlling a computer-controlled device. [Figure 5] FIG. 1 is a schematic diagram of an example of an embodiment of a control method for generating control signals for a computer-controlled device. [Figure 6] FIG. 1 is a schematic diagram of an example of an embodiment of a training method for training a neural network for use in controlling a computer-controlled device. [Figure 7a] 1 is a schematic diagram of a computer-readable medium having a writable portion containing a computer program according to one embodiment; [Figure 7b] FIG. 1 is a schematic diagram of a processor system according to one embodiment.

[0023] List of reference numbers for Figures 1 to 4, 7a and 7b The following list of reference symbols and abbreviations is provided to facilitate interpretation of the drawings and should not be construed as limiting the scope of the claims. 100 Controllers 110 Computer-controlled equipment 111 Machine Learning-Enabled Systems 120 Sensor Input Interface 122 Sensor System 130 Neural Network Storage and Neural Networks 150 processor system 152 Neural Network Unit 154 Projection Unit 156 training units 160 Control Interface 170 Training set 200 Physical Systems 210 Moving parts 212 Actuator 214 Sensors 220 Controller 230 Camera 240 Robot Arm 241 Links 242 Tools 243 Joints 300 Controller 330 processor 340 memory 350 Communication Interface 1000 Computer Readable Medium 1010 Writable area 1020 Computer Program 1110 Integrated Circuits 1120 Processing Unit 1122 memory 1124 dedicated integrated circuits 1126 Communication Elements 1130 Interconnection 1140 Processor System DETAILED DESCRIPTION OF THE INVENTION

[0024] Detailed Description of the Embodiments The subject matter of the present disclosure may be embodied in many different forms, one or more specific embodiments of which have been shown in the drawings and described in detail herein. It should be understood that the present disclosure is to be considered as an exemplification of the principles of the subject matter of the present disclosure, and is not intended to be limited to the specific embodiments shown and described. For purposes of understanding, the elements of the embodiments are described below in operation. However, it will be apparent that each element is arranged to perform the functions described as being performed by those elements.

[0025] Furthermore, the subject matter of the present disclosure is not limited to the embodiments only, but also includes all other combinations of features described in this specification or in mutually different dependent claims.

[0026] FIG. 1 illustrates a schematic diagram of an embodiment of a machine learning-enabled system 111. The system 111 includes a controller 100 and a computer-controlled device 110. For example, the controller 100 can be configured to generate control signals for the device 110. The device can include multiple interacting parts, which can interact in a complex, e.g., nonlinear, manner, e.g., in a so-called dynamical system. While the dynamical system may be explicitly modeled, this is not required. For example, the dynamical system may have nonlinear or time-varying parameters. Such dynamical systems are difficult to control, e.g., reliably control to a desired state or follow a desired trajectory. While neural networks can be trained to successfully control such systems, neural networks have the drawback that their solutions do not provide guarantees. While neural networks are often significantly more efficient than, for example, traditional controllers, they can have unexpected drawbacks. For example, a neural network may suddenly become unstable when faced with a state or sensor input that is even slightly different from that known from training. Clearly, in practical applications, such a situation is undesirable. In one embodiment, an efficient controller is provided that provides guarantees about its stability.

[0027] For example, in one embodiment, the device includes a so-called cart-pole. In this case, the parts may include the cart and the pole. The control signals may include the position, speed, and / or direction in which the cart is steered. The cart and the pole have a complex interaction and can be considered as a dynamic system.

[0028] The computer-controllable device 110 can be any device whose control is at issue. For example, the device can include one or more of a robot, a vehicle, a household appliance, a power tool, manufacturing equipment, a drone, etc. For example, a drone can be controlled to remain stable at a particular point in the air, to remain on a two-dimensional plane, or to follow a particular trajectory. For example, in the case of a vehicle, the controller can be configured for cruise control, e.g., to maintain the vehicle at a predetermined speed, or for trajectory assistance, such as lane assistance. For example, the control signals can control speed and steering to maintain the vehicle's position within its lane.

[0029] The device 110 is computer-controllable by the controller 100. The device 110 may include a further computer for controlling the device 110, for example, depending on stable control signals of the controller 100.

[0030] For example, controller 100 can include a control interface 160 configured to send stable control signals to computer-controllable device 110 for controlling computer-controllable device 110. This control can be direct, e.g., the output of controller 100 directly controls, e.g., an actuator of device 110, or indirect. For example, the control signal can be a control signal to increase force by a certain amount at a certain point, e.g., at a rotor. In such cases, device 110 can include a further controller to convert the control signal coming from controller 100 into a format suitable for device 110, e.g., for an actuator of device 110.

[0031] The system 111 may include a sensor system 122 configured to sense the computer-controlled device 110. The controller 100 may include a sensor input interface 120 configured to receive sensor signals from the sensor system 122. For example, the sensor system 122 may include one or more of an image sensor, a radar, a lidar, and a pressure sensor. The sensor signals may include one-dimensional sensor signals, such as a pressure sensor configured to measure pressure at a specific point. The sensor signals may include two-dimensional sensor signals, such as an image sensor. For example, one or more cameras may record the position of a robot arm or the position of a vehicle within a lane. Interestingly, such high-dimensional sensor signals may be directly used by the controller 100.

[0032] In one embodiment, the controller 110 receives the sensor signal and calculates an updated value for the next action, e.g., actuator input, from the sensor signal. For control purposes, the controller 110 can be repeatedly applied to the current sensor signal obtained from the sensor system 122 to repeatedly obtain a stable control signal for controlling the computer-controllable device 110. For example, the action to be performed can be discretized. The frequency at which the action is updated can vary depending on the embodiment. For example, the frequency at which the action is updated can be multiple times per second, e.g., 10 times per second or more, if the dynamic system is rapidly changing or if external disturbances may change rapidly. The action can also be updated more slowly, e.g., once per second or once every two seconds.

[0033] In one embodiment, the controller 100 can be configured to calculate a control signal to bring the state of the device 110 to a desired state. For example, the state can be a predefined state, such as a zero state or a rest state. In one embodiment, the controller is configured to receive, for example, obtain, a target state, in which case the controller 100 is configured to cause the device to reach the target state. The target state can be generated by the controller 110, for example, to cause the state of the device to follow a trajectory. The controller can include a target state interface, for example, to receive the target state from a further computer or an operator.

[0034] In one embodiment, the controller 100 can be configured to calculate control signals to cause some state of the device 110 to become a desired state. For example, this state could be the position and velocity of a vehicle, e.g., an automobile, while the control signals could be configured to maintain speed but not affect steering, e.g., in the case of cruise control. For example, in the case of a drone, the control signals could keep the drone stable in a two-dimensional plane without further restricting movement in the plane. However, in one embodiment, all states of the device may be controlled by the controller.

[0035] In one embodiment, the controller 100 is configured to vary the target state along a trajectory so that the device follows the trajectory. For example, the controller 100 can be used to steer a vehicle along a trajectory, or to move a robotic arm along a trajectory. For example, the action can be updated with a first frequency f1 relative to the current sensor signal, while the target state is updated with a second frequency f2 relative to the current target state. These two frequencies can be equal or different depending on the application. In one embodiment, f1>f2.

[0036] FIG. 2a illustrates a schematic diagram of an embodiment of a computer-controlled device 200, e.g., a physical system. A moving part 210 is shown, which may be any known moving part, such as a joint, linkage, block, axle, wheel, gear, etc. In particular, the moving part may include multiple joints. The device may be configured with actions that can be performed on the device. For example, the actions may include rotation and / or translation of the joint. The joint may be connected to another moving part, e.g., a linkage. For example, the joint may be linear, e.g., a slider, that only translates; cylindrical, e.g., that rotates and slides; or rotary, e.g., that only rotates.

[0037] 2 also shows an actuator 212 and a controller 220, e.g., an embodiment of the controller 100. The actuator 212 is configured to receive control signals from the controller 220 to cause changes in position parameters of the moving parts. For example, some parts can be displaced or rotated, etc. A sensor system 214, e.g., sensors 214, provides feedback regarding the state of the device. For example, the sensors 214 can include one or more image sensors, force sensors, etc.

[0038] For example, the device 200 may be a robot, a vehicle, a household appliance, an autonomous vehicle, a power tool, a manufacturing device, etc. The device 200 is at least partially under computer control. For example, the device 200 may be configurable to transition to a particular goal state. The device 200 may have a mode that is, for example, fully or partially controlled by an operator, e.g., a human. A controller 220 may be used to control the system. The planning task may be continuous, discrete, or discretized. The observations may be continuous and high-dimensional, e.g., image data.

[0039] For example, the device may be a vehicle, e.g., an automobile, configured for autonomous driving. Typically, an automobile performs continuous observations, e.g., distance to an exit, distance to a preceding vehicle, exact position within a lane, etc. The vehicle's controller can be configured to decide how to act and where to drive, e.g., change to another lane. For example, sensors can be used to learn the state of the automobile in its environment, while possible driving actions the automobile can take affect the state of the automobile.

[0040] For example, the device may be a manufacturing device. For example, the manufacturing device may be configured to have a set of continuous or discrete actions, possibly with continuous parameters. For example, the actions may include a tool to be used and / or parameters for the tool, such as a drilling action at a specific position or angle. For example, the manufacturing device may be a computer numerically controlled (CNC) device. A CNC device may be configured for automatic control of machining tools, such as drills, boring tools, lathes, 3D printers, etc., by a computer. A CNC device may machine a piece of material, such as metal, plastic, wood, ceramic, composites, etc., to meet specifications.

[0041] The observable state of a manufacturing device is often continuous and often noisy, e.g., a noisy drilling environment, a noisy manufacturing part environment, etc. An embodiment can be configured to plan how to create a part, e.g., first drill in place, then perform some secondary action.

[0042] In one embodiment, the controller is configured to steer the device 200 into a rest state, e.g., a state suitable for powering off, e.g., a state in which the device is safe for an operator to access, e.g., for maintenance purposes.

[0043] FIG. 2b schematically illustrates an example of an embodiment of a robotic arm 240. In one embodiment, the apparatus includes a robotic arm. FIG. 2b illustrates an example of a robotic arm. For example, the robotic arm may include one or more links 241, a tool 242, and a joint 243. The robotic arm may be controllable, for example, by a controller 220, to move the joints. The robotic arm 240 may be associated with one or more sensors, some of which may be integrated into the arm. The sensor may be a camera, for example, camera 230. The sensor output, optionally including an image, may be an observable state of the robotic arm. In one embodiment, the controller is configured to steer the robotic arm, such as to a rest state.

[0044] Even if the observations may be continuous and high-dimensional, e.g., the state of all joints, sensor measurements such as camera images, etc., the controller can calculate safe actions to reach a desired state and / or follow a desired trajectory.

[0045] Returning to Figure 1, the controller 100 may include a processor system 150 configured to apply a neural network and a projection layer to the sensor signals.

[0046] For purposes of explanation, the neural network and the projection layer are shown and described separately, but it is understood that in one embodiment, the neural network may have multiple layers, one of which is a projection layer. Typically, the projection layer will be the final layer in such a neural network, although this is not strictly required. For example, further processing may be performed on the output of the neural network, such as transforming from one domain to another, for example, scaling the signal, transforming the coordinate system, etc. Preferably, the further processing preserves the improved stability provided by the projection layer.

[0047] For example, the system 111 may include a neural network storage configured to store the trained neural network 130. Interestingly, the neural network 130 defines raw controls for the computer-controlled device 110. For example, the controls are functions that map sensor signals to actions. If the neural network 130 were trained without a projection layer, the output of the neural network would be used as the control signal. However, such a controller may fail if the device or sensor values ​​enter unexpected regions.

[0048] The neural network storage may or may not be part of the controller 100. For example, the parameters of the neural network may be stored externally, e.g., in cloud storage, or may be stored locally, e.g., in local storage, e.g., memory, hard drive, etc.

[0049] A large set of potential stable measures can be defined for a dynamical system. For example, these measures are guaranteed to reach a particular desired target state, e.g., the origin, over time. Such stability is usually defined under some assumption about the disturbances that can be expected in the system. Clearly, stable measures cannot exist in all cases if external disturbances are allowed to be unbounded. However, it is possible to define stable measures, for example, under normative constraints on external disturbances. While stable measures have the advantage of providing guaranteed stability, e.g., reaching a particular target state at a certain time or arbitrarily approaching a particular target state, there is usually no indication as to which stable measures efficiently provide such an advantage. The system 150 advantageously includes a projection function that maps inputs to a space of stable measures defined for the system, e.g., the dynamical system. The projection function maps the space of raw control measures to the space of stable control measures of predefined stable control measures. Thus, by applying the projection, raw control signals are mapped to stable control signals.

[0050] Thus, the neural network is still free to learn any raw control signal it deems appropriate, but the output of the neural network is passed through a projection function, e.g., as contained in a projection layer. A neural network is obtained that has a large parameter space and efficient neural network training, but is guaranteed to provide a control strategy that satisfies certain conditions, such as a stability criterion. Essentially, a neural network is obtained that can freely learn any control strategy, as long as the control strategy is stable.

[0051] For example, processor 150 can be configured to include a neural network unit 152 and a projection unit 154. For example, neural network unit 152 can be configured to apply neural network 130 to the sensor signals to obtain raw control signals. Note that if neural network 130 is trained with a projection layer, the raw control signals cannot be directly applied to device 110. Rather, the raw control signals can be viewed as indexes, or pointers, into a space of stable measures.

[0052] While the neural network unit 152 and the projection unit 154 can be configured for a fixed target state, such as a resting state, the neural network unit 152 and the projection unit 154 can alternatively be configured to receive a target state as an input and configured to control the computer-controllable device 110 to reach the target state. For example, the target state can be a parameter of a projection function, but can also be a neural network input for the neural network 130.

[0053] The sensor signal inputs for the neural network 130 and the raw control signal outputs of the neural network 130 may be represented, for example, as vectors containing real numbers. For example, the values ​​may be scalable, for example, between 0 and 1. For example, the neural network may include known neural network layers such as convolutional layers, ReLu layers, pooling layers, etc. The dimensionality of the raw control signals may be the same as the dimensionality of the stable control space. Typically, the dimensionality of the raw control signals may be the same as the dimensionality of the output of the projection function. For example, in one embodiment, the input to the neural network includes image data and the neural network includes a convolutional layer.

[0054] The output of the projection function represents a stable control signal, which may represent, for example, an input for the computer-controlled device 110. For example, the stable control signal may be a vector. The elements of the vector may be relative actions, such as increasing a distance, voltage, angle, etc. The elements of the vector may be absolute actions, such as setting a speed or distance to a particular value.

[0055] A projection function can be constructed to map a control strategy, e.g., any control strategy, into a space of stable control strategies. Such a space can be defined in different ways. For example, a stability equation can be imposed on a stable control strategy. For example, a Lyapunov function can be required to be decreasing for a stable control strategy. A projection can be viewed as a piecewise continuously differentiable function, mapping from one space to another. This means that a learning algorithm can propagate through the projection function.

[0056] Depending on the dynamics of the device, different projection layers are possible. One example of a projection layer is to limit the control signals to safe values. For example,

number

number

number

number

number

number

number

[0057] In another example, the projection function can solve an optimization problem whose solution is piecewise differentiable with respect to the raw control signals. In general, the projection function can be selected with respect to a dynamical system. The dynamical system can define a set of stable control measures, as in control theory, for example. A neural network can be trained to select an appropriate one of the stable control measures to reduce the loss function.

[0058] FIG. 3 schematically illustrates an example of an embodiment of a controller 300. For example, the controller 300 of FIG. 3 can be used to control the device 110. The controller 300 may include a processor system 330, a memory 340, and a communication interface 350. The controller 300 may communicate with the device 110, external storage, input devices, output devices, and / or one or more sensors via a computer network. The computer network may be the Internet, an intranet, a LAN, a WLAN, or the like. The computer network may be the Internet. The system may include a connection interface arranged to communicate with the interior or exterior of the system as needed. For example, the connection interface may include a connector, such as a wired connector, e.g., an Ethernet connector, an optical connector, or the like, or may include a wireless connector, e.g., an antenna, e.g., a Wi-Fi antenna, a 4G antenna, or a 5G antenna.

[0059] The implementation of a controller, e.g., controller 100, controller 300, etc., can be implemented in a processor system, e.g., one or more processor circuits, e.g., microprocessors, examples of which are shown herein. FIGS. 1 and 3 illustrate possible functional units of a processor system. For example, these figures can be used as blueprints of a possible functional organization of a processor system. In these figures, the processor circuits are not illustrated separately from the units. For example, the illustrated functional units can be implemented in whole or in part in computer instructions stored in the controller, e.g., in electronic memory, and executable by the controller's microprocessor. In hybrid embodiments, the functional units are implemented partly in hardware, e.g., as a coprocessor, e.g., a neural network coprocessor, and partly in software stored and executed on the controller. Parameters of the network and / or training data can be stored locally or in cloud storage.

[0060] 4 shows a schematic diagram of an example of an embodiment of a training system 101 for training a neural network for use in controlling a computer-controlled device. For example, the training system 101 can be used to train a neural network 130 used in a controller 100. The controller 300 can be adapted to implement the training system 101 in the same manner as or in place of the controller 100, as needed, for example, by providing access to training data and / or software configured to implement the training functions.

[0061] For example, the training system 101 may include a training data interface for accessing training data. For example, the training data may include sensor signals representative of a sensor system sensing a computer-controlled device. The sensor data may be obtained, in whole or in part, from the actual device 110, e.g., by measurements using the sensor system. The sensor data may be obtained, in whole or in part, by a simulation, e.g., a simulation of the device 110.

[0062] System 101 may include neural network storage configured to store a neural network that defines raw control measures for a computer-controlled device, such as storage within controller 100. Training system 101 is configured for the neural network and projection function, such as controller 100. For example, system 101 may include neural network unit 152 and projection unit 154 as within controller 100, except that the neural network of neural network unit 152 has not yet been fully trained.

[0063] The training system 101, e.g., a processor system of the training system 101, can be configured to, for example, apply a neural network to sensor signals obtained from the training data 170 to obtain a neural network output, e.g., a raw control signal, and apply a projection function to the neural network output to obtain a stable control signal. Thanks to the projection layer, the resulting control signal becomes stable, but this does not mean that it is always good. Therefore, a loss function can be applied to the stable control signal to obtain a value that represents the goodness of the control measure defined by the combination of the neural network and the projection layer. For example, the device can be simulated for a predetermined period of time to determine progress toward a goal state. During that period, actions may or may not be updated. Neural network parameters are then modified to improve the loss function. For example, backpropagation can be applied to the loss function, the projection layer, and the neural network.

[0064] Embodiments of system 100 or 101 may include a variety of selectable communication interfaces. For example, the interface may be a network interface to a local area network or a wide area network, such as the Internet, a storage interface to internal or external data storage, a keyboard, an application program interface (API), etc. Systems 100 and 101 may have a user interface, which may include well-known elements such as one or more buttons, a keyboard, a display, a touchscreen, etc. The user interface may be configured to accommodate user interaction for configuring the system, training the network against a training set, or applying the system to new sensor data, setting target states, setting trajectories, etc.

[0065] The storage can be implemented as electronic memory, such as flash memory, or as magnetic memory, such as a hard disk. The storage can include multiple discrete memories that together constitute the storage. The storage can include temporary memory, such as RAM. The storage can be cloud storage. The storage can be used to store neural networks, software, training data, etc.

[0066] Systems 100 and / or 101 can be implemented in a single device or, possibly, in a distributed system. Typically, systems 100 and 101 include a microprocessor executing appropriate software stored on the system, which may be downloaded and / or stored in corresponding memory, e.g., volatile memory such as RAM or non-volatile memory such as flash. Alternatively, the system can be implemented, in whole or in part, in the form of programmable logic, e.g., a field programmable gate array (FPGA). The system can be implemented, in whole or in part, as a so-called application-specific integrated circuit (ASIC), e.g., an integrated circuit (IC) customized for a particular application. For example, the circuit can be implemented in CMOS using a hardware description language such as Verilog, VHDL, etc. In particular, systems 100 and 101 can include circuitry for evaluating neural networks.

[0067] The processor circuitry may be implemented in a distributed manner, for example as multiple sub-processor circuits. The storage may be distributed across multiple distributed sub-storages. Some or all of the memory may be electronic memory, magnetic memory, etc. For example, the storage may have volatile and non-volatile portions. Some of the storage may be read-only.

[0068] Below are some further optional refinements, details and embodiments.

[0069]

number

number

number

number

number

[0070] In one embodiment, learning a provably robust nonlinear controller using reinforcement learning can be accomplished as follows: A linear, e.g., neural network-based, vehicle class parameterized by θ

number

number

number

number

number

[0071] The projection operator can be implemented in a differentiable way, for example, by using differentiable convex optimization layers, see for example the paper "Differentiable Convex Optimization Layers" by A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Zico Kolter. Therefore, to optimize this problem, we use a neural network

number

[0072] In addition to designing for stability, it is often desirable to design a controller that optimizes some performance objective. Different classes of objectives can be used in different settings. For example, in the H∞ paradigm, the objective is to minimize the ratio between the L2 norm of some system output and the L2 norm of the (potentially unbounded) disturbance, i.e., the L2 gain of the disturbance-to-output mapping. Time-dependent signals

number

number

number

number

number

[0073] Equation 4 defines the solution to the LQR problem. Given some specific dynamics description, e.g., norm-bounded LDI or polytopic, this optimization can be solved. In this way, the optimal linear measures and Lyapunov function can be obtained. If we have a Lyapunov function, this Lyapunov function defines a set of stable measures that includes the optimal linear measures but also nonlinear measures. The raw control output of the neural network can be projected onto this set of measures that stabilizes the system, as qualified by the Lyapunov function obtained, for example, by solving equation (4).

[0074] We consider controlling nonlinear continuous-time dynamical systems of the form shown in equation (1). Often, we consider the dynamics as an alternative (potentially time-varying) linearized form

number

number

[0075] Norm-bounded LDI In this setting, the dynamics are

number

number

[0076] Assuming a time-invariant linear control measure u(t) = Kx(t), the specification of a set of measures that will stabilize the system under this class of worst-case perturbations can be formulated via a set of linear matrix inequalities. At a high level, these methods involve specifying the controller gain K and the quadratic Lyapunov function V(x) = x such that the Lyapunov function guarantees exponential stability of the resulting system under any disturbance subject to the norm bound (7). T Px and Px, where exponential stability is achieved by satisfying the following condition:

number

[0077] The following linear matrix inequality

number

number

[0078] Polytopic LDI Another interesting setup is the polytopic LDI (PLDI) setup, where the dynamics is

number

number

[0079] As above, the variable

number

number

[0080] As mentioned before, the resulting controller and quadratic Lyapunov function are: K=YS -1 and P=S -1 The above equation again gives a convex set C of linear stabilizing controllers that can be used, for example, in equation (4). l to provide.

[0081] Nonlinear control measures, potentially parameterized by deep neural networks, can be obtained that are guaranteed to obey the equivalent stability conditions imposed by the robust specifications exemplified above. In general, it is difficult to describe conditions that globally characterize the stability of a nonlinear controller, but sufficient conditions for stability can be created, for example, by ensuring that a stability criterion holds, e.g., by ensuring that a given measure reduces the Lyapunov function given the robustness specifications.

[0082] Nominal nonlinear controller output

number

[0083] Stability in NLDI To guarantee exponential stability for NLDI, for any perturbation within the admissible set (7), the control means must

number

number

number

number

[0084] Therefore, C NLDI (P,α,x) is the sufficient reduction criterion

number

number

number

[0085] This projection is then used in the neural network optimization loop (2), i.e., Cn = C NLDI (P,α,x). This projection does not necessarily have a closed form, but can be implemented by optimization, for example, using differentiable convex optimization layers from the convex optimization library CVXPY. See, for example, "Differentiable Convex Optimization Layers" by A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Zico Kolter.

[0086] Stability in NLDI when D=0 The case where D=0, i.e., the norm bounded by the disturbance w, does not depend on the control measures, is a special case of the above problem. This form of NLDI occurs in many general settings, for example, when the linearization error is characterized by w, but the original dynamics only linearly depend on the controller. In this case, the projection of the nominal measures onto the set of stabilizing controllers can be derived in closed form. Under this set form, we first find the C of the stabilizing controllers NLDI-0 (P,α,x) is presented. In this case, C NLDI-0 Let (P,α,x) be the given state

number

number

[0087] The above inequality represents a linear constraint, so CNLDI-0 (P,α) is a convex set in π(x).

[0088] Interestingly, by solving the optimization problem (4) under the constraint (9) for a given value of α>0, we obtain a projection P, and then define the corresponding constraint set as

number

number

[0089] As mentioned before, this projection is then used in the neural network optimization (2), i.e., Cn = C NLDI-0 It can be used by (P,α,x).

[0090] Stability in PLDI Similarly, to ensure exponential stability for PLDI, for any (A(t), B(t)) within the admissible set (11), the control means

number

number

number

number

[0091] Therefore, C PLDI (P,α,x) is the sufficient reduction criterion

number

[0092] Below we describe two detailed embodiments that demonstrate improved stability under hostile disturbances: one in the domain of the cart-pole task and the other in the domain of the quadrotor.

[0093] Cart-pole In the Cart-Pole task, the goal is to balance an inverted pendulum resting on a cart. The state of the system is:

number

number

number

number

number

number

[0094] Planar Quadrotor In the planar quadcopter setup, the goal is to stabilize the quadcopter on a two-dimensional plane. The state of this system is

number

number

number

[0095] In both the cart-pole example and the quadrotor example, the nominal nonlinear control measures are

number

number

number

[0096] The robust neural network-based method is compared to a robust (linear) LQR controller, a non-robust neural network trained via direct means search, and a standard non-robust (linear) LQR controller. Performance is evaluated both in the original setting (e.g., under the original dynamics) and under a test-time disturbance w(t) that is adversarially generated to minimize the reduction of the Lyapunov function (see Appendix 7). All methods are evaluated over a 1-second period with a discretization of 0.005 seconds.

[0097] Table 1 shows the performance of these methods in the domain of interest. An "X" is marked to indicate cases where the quadratic loss integral over a given period in a test set of states is reported, or where the associated method becomes unstable (hence the loss is inf, NaN, or orders of magnitude larger than other approaches). These results demonstrate the fundamental benefits of the robust NN approach. In all cases, the performance of the robust NN on the original dynamics (the goal for which optimization was performed) improves over that of the robust LQR approach (e.g., a linear controller that also provides stability guarantees). At the same time, since both the traditional (non-robust) LQR method and the non-robust NN are optimizing for expected performance rather than worst-case performance, performance often improves within the range of the original nominal dynamics, as expected. The original nominal dynamics are denoted by O, and the adversarial dynamics are denoted by A. [Table 1]

[0098] However, non-robust LQR and neural network approaches can diverge or perform significantly worse, especially when adversarial perturbations are applied that are still within acceptable norm bounds and therefore valid perturbations. In contrast, both robust neural networks and robust LQR approaches are stable under such perturbations.

[0099] Both the robust and non-robust NN approaches converge very quickly to their final performance levels. Non-robust NNs often become unstable under adversarial dynamics very early in the process. Overall, these results demonstrate that the present embodiment can learn more expressive measures than traditional robust LQR while ensuring that these measures are stable.

[0100] 5 shows a schematic diagram of an example of an embodiment of a control method (500) for generating control signals for a computer-controlled device. The method 500 is computer-implementable. The computer-controlled device includes multiple parts interacting in a dynamic system. The control method includes: receiving (510) a sensor signal from a sensor system sensing the computer-controlled device, the sensor signal being indicative of a current state of the computer-controlled device; applying (520) a neural network to the current sensor signals, the neural network being configured to define raw control measures for the computer-controlled device and map the sensor signals to raw control signals; applying (530) a projection function to the raw control signals that maps the space of raw control measures to a space of stable control measures of a predefined stable control measure, thereby obtaining stable control signals; causing the stable control signal to control a computer-controllable device (540); may include:

[0101] Figure 6 shows a schematic diagram of an example embodiment of a training method (600) for training a neural network for use in controlling a computer-controlled device. The computer-controlled device includes multiple parts that interact in a dynamical system. Method 600 is computer-implementable. The training method includes: receiving (610) a sensor signal representative of a sensor system sensing a computer-controlled device, the sensor signal indicating a current state of the computer-controlled device; applying (620) a neural network to the current sensor signals, the neural network being configured to define raw control measures for the computer-controlled device and map the sensor signals to raw control signals; applying (630) a projection function to the raw control signals that maps the space of raw control measures to a space of stable control measures of a predefined stable control measure, thereby obtaining stable control signals; Calculating (640) a loss associated with a stable control signal and training parameters of the neural network such that the loss is reduced; may include:

[0102] For example, the control and training methods may be computer-implemented methods. For example, accessing training data and / or receiving input data may be performed using a communications interface, e.g., an electronic interface, a network interface, a memory interface, etc. Storing or retrieving parameters, e.g., parameters of a network, may be performed from electronic storage, e.g., from a memory, a hard drive, etc. For example, applying a neural network to data from the training data and / or adjusting the stored parameters to train the network may be performed using an electronic computing device, e.g., a computer.

[0103] A neural network being trained and / or applied can have multiple layers, which can include, for example, convolutional layers. For example, a neural network can have at least 2, 5, 10, 15, 20, or 40 or more hidden layers. The number of neurons in a neural network can be, for example, at least 10, 100, 1000, 10,000, 100,000, 1,000,000, or more.

[0104] As will be apparent to those skilled in the art, many different ways of implementing the method are possible. For example, although the steps may be performed in the order shown, the order of the steps may be changed or some steps may be performed in parallel. Furthermore, other method steps may be inserted between steps. The inserted steps may be modifications of the methods described herein or may be unrelated to the method. For example, some steps may be performed at least partially in parallel. Furthermore, a given step need not be completed before the next step begins.

[0105] Embodiments of the methods may be implemented using software including instructions for causing a processor system to perform methods 500 and / or 600. The software may include only steps performed by a particular sub-entity of the system. The software may be stored on a suitable storage medium such as a hard disk, floppy, memory, optical disk, etc. The software may be transmitted as a signal via wire or wirelessly, or using a data network, e.g., the Internet. The software may be available for download and / or for remote use on a server. Embodiments of the methods may be implemented using a bitstream arranged to configure programmable logic, e.g., a field programmable gate array (FPGA), to perform the methods.

[0106] It will be understood that the subject matter of the present disclosure also extends to computer programs, particularly computer programs on or in a carrier, adapted to carry out the subject matter of the present disclosure. The program may be in the form of object code, such as source code, object code, code intermediate source, and partially compiled form, or any other form suitable for use in implementing embodiments of the present methods. One embodiment of a computer program product includes computer-executable instructions corresponding to each of the processing steps of at least one of the described methods. These instructions may be subdivided into subroutines and / or stored in one or more files, which may be statically or dynamically linked. Another embodiment of a computer program product includes computer-executable instructions corresponding to each of the devices, units, and / or parts of at least one of the described systems and / or products.

[0107] FIG. 7a illustrates a computer-readable medium 1000 having a writable portion 1010 containing a computer program 1020, the computer program 1020 including instructions for causing a processor system to implement a control method and / or a training method according to one embodiment. The computer program 1020 may be embodied on the computer-readable medium 1000 as a physical mark or by magnetization of the computer-readable medium 1000. However, any other suitable embodiment is contemplated. Furthermore, while the computer-readable medium 1000 is illustrated herein as an optical disk, it will be understood that the computer-readable medium 1000 may be any suitable computer-readable medium, such as a hard disk, solid-state memory, flash memory, etc., and may be non-recordable or recordable. The computer program 1020 includes instructions for causing a processor system to implement the control method and / or training method described above.

[0108] FIG. 7b shows a schematic diagram of a processor system 1140 according to one embodiment of a controller and / or training system or device. The processor system includes one or more integrated circuits 1110. FIG. 7b shows a schematic diagram of the architecture of the one or more integrated circuits 1110. The circuit 1110 includes a processing unit 1120, e.g., a CPU, for executing computer program components to perform a method according to one embodiment and / or to implement a module or unit thereof. The circuit 1110 includes a memory 1122 for storing programming code, data, etc. A portion of the memory 1122 may be read-only. The circuit 1110 may include a communication element 1126, e.g., an antenna, a connector, or both. The circuit 1110 may include a dedicated integrated circuit 1124 for performing some or all of the processing defined in the method. The processor 1120, the memory 1122, the dedicated IC 1124, and the communication element 1126 may be connected to each other via an interconnect 1130, e.g., a bus. The processor system 1110 can be configured for contact and / or contactless communication using antennas and / or connectors, respectively.

[0109] For example, in one embodiment, the processor system 1140, e.g., a controller and / or training system / device, may include a processor circuit and a memory circuit, where the processor is configured to execute software stored in the memory circuit. For example, the processor circuit may be an Intel Core® i7 processor, an ARM Cortex-R8, or the like. In one embodiment, the processor circuit may be an ARM Cortex M0. The memory circuit may be a ROM circuit or a non-volatile memory, e.g., flash memory. The memory circuit may be a volatile memory, e.g., SRAM memory. In the latter case, the device may include a non-volatile software interface, e.g., a hard drive, a network interface, or the like, configured to provide the software.

[0110] The processor system may include multiple microprocessors, each configured to independently perform the methods described herein, or configured to perform the steps or subroutines of the methods described herein such that the processors cooperate to achieve the functionality described herein. Furthermore, when the device and / or system is implemented in a cloud computing system, different hardware components may reside in separate devices. For example, the processors may include a first processor in a first server and a second processor in a second server.

[0111] It should be noted that the above embodiments are illustrative rather than limiting of the subject matter of the present disclosure, and that those skilled in the art will be able to design numerous alternative embodiments.

[0112] In the claims, reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The use of phrases such as “at least one,” when preceding a list of elements, denotes selection of all or any subset of the elements from that list. For example, the phrase “at least one of A, B, and C” should be understood to include A only, B only, C only, both A and B, both A and C, both B and C, or all of A, B, and C. The subject matter of the present disclosure can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a device claim enumerating several elements, several of these elements may be embodied by a single and identical item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0113] In the claims, reference signs in parentheses refer to the reference signs in the drawings of exemplary embodiments or modes of embodiment, thus facilitating the comprehension of the claims, and should not be construed as limiting the scope of the claims.

Claims

1. A controller (100) for generating control signals for a computer-controlled device (110) comprising a plurality of interacting parts (210), comprising: a sensor input interface (120) configured to receive sensor signals indicative of a current state of the computer-controlled device (110) from a sensor system (122) sensing the computer-controlled device (110); a neural network storage (130) configured to store a trained neural network, said neural network defining raw control means for said computer-controlled device (110); A processor system (150), comprising: applying the neural network configured to map the sensor signal to a raw control signal to the current sensor signal; applying a projection function to the raw control signals, which maps the space of raw control means onto a space of predefined control means, thereby obtaining control signals for controlling the computer-controlled device (110); The control signal controls the computer-controlled device (110). a processor system (150) configured as follows: A controller (100) comprising: The projection function is a function of the value for the raw control signal [Equation 1] and reducing the value of the raw control signal if the value for the raw control signal exceeds a threshold (η); The projection function is [Equation 2] is defined as where x is the sensor input and π(x) is the control signal for controlling the computer-controlled device (110); [Equation 3] is the raw control signal, ρ is a vector, and η is a value, A controller (100) comprising:

2. the neural network is repeatedly applied to current sensor signals obtained from the sensor system (122) to repeatedly obtain the control signals for controlling the computer-controlled device (110); The controller of claim 1 .

3. the sensor system (122) includes one or more of an image sensor, a radar, a lidar, and a pressure sensor; 3. The controller according to claim 1 or 2.

4. The neural network is further configured to receive a target state as an input and to control the computer-controlled device (110) to reach the target state. A controller according to any one of claims 1 to 3.

5. the plurality of parts interacting according to a single dynamic system; A controller according to any one of claims 1 to 4.

6. the computer-controlled device (110) comprises one or more of a robot (240), a vehicle, a household appliance, a power tool, a manufacturing device, or a drone; A controller according to any one of claims 1 to 5.

7. The space of the control means is defined by a stability equation: A controller according to any one of claims 1 to 6.

8. the raw control signal and the control signal are represented as vectors; The elements of said vector represent said control signals which represent inputs for said computer-controlled device (110); A controller according to any one of claims 1 to 7.

9. 5. Raw signal [Equation 4] is a linear controller (Kx) and a neural network [Equation 5] is the sum of A controller according to any one of claims 1 to 8.

10. A control method (500) for generating control signals for a computer-controlled device (110) comprising multiple parts interacting in a dynamic system, comprising: receiving (510) a sensor signal indicative of a current state of the computer-controlled device (110) from a sensor system (122) sensing the computer-controlled device (110); applying (520) to the current sensor signals a neural network configured to define raw control means for the computer-controlled device (110) and map the sensor signals to raw control signals; applying (530) a projection function to the raw control signals, which maps the space of raw control means onto a space of predefined control means, thereby obtaining control signals for controlling the computer-controlled device (110); causing the control signal to control (540) the computer-controlled device (110); A control method (500) comprising: The projection function is a function of the value for the raw control signal [Equation 6] and reducing the value of the raw control signal if the value for the raw control signal exceeds a threshold (η); The projection function is [Equation 7] is defined as where x is the sensor input and π(x) is the control signal for controlling the computer-controlled device (110); [Equation 8] is the raw control signal, ρ is a vector, and η is a value, A control method (500) comprising:

11. 1. A training method (600) for training a neural network for use in controlling a computer-controlled device (110) comprising multiple parts interacting in a dynamical system, comprising: receiving (610) a sensor signal representative of a sensor system (122) sensing the computer-controlled device (110), the sensor signal indicating a current state of the computer-controlled device (110); applying (620) to the current sensor signals a neural network configured to define raw control means for the computer-controlled device (110) and map the sensor signals to raw control signals; applying (630) a projection function to the raw control signals that maps the space of the raw control means onto a space of predefined control means, thereby obtaining control signals for controlling the computer-controlled device (110); Calculating (640) a loss associated with the control signal and the training parameters of the neural network such that the loss is reduced; A training method (600) comprising: The projection function is a value for the raw control signal. [Equation 9] and reducing the value of the raw control signal if the value for the raw control signal exceeds a threshold (η); The projection function is [Equation 10] is defined as where x is the sensor input and π(x) is the control signal for controlling the computer-controlled device (110); [0011] is the raw control signal, ρ is a vector, and η is a value, A training method (600) comprising:

12. 1. A training system for training a neural network for use in controlling a computer-controlled device (110) comprising multiple parts interacting in a dynamical system, comprising: a training data interface for receiving sensor signals representative of a sensor system (122) sensing the computer-controlled device (110), the sensor signals indicating a current state of the computer-controlled device (110); a neural network storage (130) configured to store a neural network defining a raw control strategy for said computer-controlled device (110); A processor system (150), comprising: applying a neural network to the current sensor signals, the neural network being configured to define raw control means for the computer-controlled device (110) and to map the sensor signals to raw control signals; applying a projection function to the raw control signals, which maps the space of raw control means onto a space of predefined control means, thereby obtaining control signals for controlling the computer-controlled device (110); Calculating a loss associated with the control signal and the training parameters of the neural network such that the loss is reduced. A processor system (150) configured as follows: In a training system comprising: The projection function is a value for the raw control signal. [0012] and reducing the value of the raw control signal if the value for the raw control signal exceeds a threshold (η); The projection function is [0013] is defined as where x is the sensor input and π(x) is the control signal for controlling the computer-controlled device (110); [0014] is the raw control signal, ρ is a vector, and η is a value, A training system characterized by:

13. A temporary or non-transitory computer-readable medium (1000) comprising data (1020) representing instructions that, when executed by a processor system (150), cause the processor system (150) to perform the method of claims 10 and / or 11.

Images