Training and use of a software agent for an optimized control of an actuator

Abstract

The invention relates to a method for training a software agent of a system for a reinforcement learning process for an optimized control of an actuator, to a corresponding software agent, to a method for parameterizing an actuator using the software agent and corresponding devices for carrying out the aforementioned method, and to a corresponding actuator. The training method has the steps of inputting (S11) a test signal and an initial parameterization into an actuator representation of the actuator, determining (S12) characteristics of the actuator representation in response to the input, inputting (S13) the characteristics into the software agent, receiving (S14) a parameterization as an output of the software agent, inputting (S15) the parameterization received as the output of the software agent into the actuator representation, and repeatedly carrying out the steps until a termination condition is reached.

Description

[0001] Training and using a software agent for optimized control of a

[0002] Actuator

[0003] Technical area

[0004] The present invention relates to a method for training a software agent of a reinforcement learning system for the optimized control of an actuator, an associated software agent, a method for parameterizing an actuator using this software agent and associated devices for carrying out the aforementioned methods and an associated actuator.

[0005] State of the art

[0006] Parameterization, i.e., the selection of suitable parameters for controlling actuators, such as electrohydraulic actuators, can be performed in various ways. If a robust model for the existing control loop is available, the respective parameters can be derived from it. However, since modeling is associated with considerable effort and is often not possible due to system complexity, for example, due to manufacturing tolerances, the parameterization of actuators, such as electrohydraulic actuators, is usually performed manually by experienced application engineers. Implementation is usually carried out using so-called basic parameters, which must then be manually adjusted in the field.

[0007] The effort required for such parameterization increases even further when the parameters vary depending on the manufacturing tolerance for the same component. This behavior can be observed, for example, in the torque modulation of electrohydraulically actuated transmissions. Due to the high variance of the required electrohydraulic actuators, actuator-specific parameterization is necessary for a stable shifting process. Since theoretical modeling is not possible for tolerance-dependent component behavior in connection with nonlinear parameter settings, the parameters are set and adjusted manually, as described above. This entails correspondingly high costs.

[0008] US 2019 / 232488 A1 describes a method for a robot parameterization process. This method awards appropriate rewards for a reinforcement learning system depending on the optimal performance of a robot's task.

[0009] Description of the invention

[0010] In a first aspect, the present invention relates to a method for training a software agent of a reinforcement learning system for the optimized control of an actuator. Reinforcement learning refers to a system in which an agent, hereinafter referred to as a software agent, selects suitable control parameters in the form of actions through interaction with an actuator, such as an electrohydraulic actuator, in order to maximize a reward. In this context, the reward describes the quality of the selected control parameters. Various methods in the field of reinforcement learning are known to those skilled in the art.What reinforcement learning methods have in common is that the agent is not shown which action is best in which situation. Instead, it receives a reward, which can also be negative, at specific times through its interaction with its environment. Through repeated interaction with the environment, the software agent can learn which changes in which parameters have which effects on the environment, and which changes in which parameters in which situation, also referred to in this context as the state, lead to a beneficial reward.Such a system for reinforcement learning with a software agent is to be trained with the method described here for controlling an actuator, such as the aforementioned electrohydraulic actuator, so that a correspondingly trained software agent can later be used to parameterize an actuator unknown to it and thus to select optimal control parameters for an actuator unknown to it.

[0011] To train the software agent, a test signal and initial parameterization are first entered into an actuator representation. A test signal can be a signal whose optimal response is known. For example, it can be a step function, where the optimal response of the system is, for example, a step response in a short time. Inputting a sine signal and determining how well the actuator output follows this sine signal is also conceivable as a test signal. Initial parameterization can be the selection of a parameter set that is preselected as the initial state. An actuator representation can be either the actuator itself or, for example, a simulated version of the actuator, i.e. a model of the actuator. For example, the behavior of an actual actuator can be incorporated into a simulation orA model can be created that, by entering a large number of or all possible input parameters and measuring the corresponding response of the actuator, maps the behavior in a digital representation of the actuator. This simulation can then be used to train the software agent, making training possible in a shorter time. A discretization of the parameterization can also be performed. The discretization is preferably chosen such that changes in the actuator input lead to distinguishable changes in the actuator output. Measurement uncertainties, in particular, can be taken into account here.

[0012] The response of the actuator representation—that is, the actuator itself or the actuator model or simulation—is then determined using suitable parameters. Such parameters can be measured values ​​that represent the behavior of the actuator. The parameters thus define a state of the actuator, which results from the input of the parameters. A corresponding discretization of the parameters is also possible here.

[0013] The specified parameters are then input into the reinforcement learning system's software agent. The software agent is programmed such that the desired behavior of the actuator's parameters for the test signal leads to an increased reward. For example, if the selected parameterization leads to a fast step response when entering a step function, a positive or increased reward is awarded. Furthermore, the software agent is programmed to output a parameterization for the actuator.

[0014] This parameterization is then received as output from the software agent.

[0015] The output of the software agent, i.e. the parameterization generated by it, is then input into the actor representation.

[0016] The aforementioned steps are then repeated several times, with the software agent attempting to increase its reward each time by adjusting the parameters accordingly. At the same time, the software agent learns which parameter changes in which state—that is, for which parameters—lead to an increased reward and thus to better control of the actuator or its representation. This leads to an optimization of the parameterization.

[0017] The aforementioned method thus allows the training of a software agent capable of parameterizing an actuator, such as an electrohydraulic actuator. The software agent created here makes it possible to individually parameterize actuators and thus improve actuator control without having to manually configure each individual actuator individually.

[0018] Thus, an agent can be trained that learns the production-specific dependency between the actuator and its controller and control parameterization.

[0019] As described above, it is possible in one aspect for the actor representation to be either the actor itself or a model or simulation of the actor. Regarding the creation of the corresponding model or simulation, reference is made to the above explanations. The use of a model or simulation of the actor allows for faster training of the software agent.

[0020] In a further aspect, the steps described above for training the software agent can be performed with different actuators or their actuator representations. This allows the agents to be trained for a variety of manufacturing tolerances and thus generalize for use with different actuators.

[0021] In one aspect, the parameterization can be at least one of a control parameter for a current controller for controlling the actuator and a control parameter for the actuator itself. Thus, in addition to the control parameters that flow directly into the actuator, for example, a dither map for controlling the actuator, the parameterization of the actuator also includes the parameterization that flows into the current controller used to control the actuator. Thus, in this case, a parameterization can, for example, also include the P, I, and D components of a PID controller in addition to the dither map.

[0022] In one aspect, for example, the directed sum of different states over time can be used to determine the reward. Thus, the reward can depend on different states and thus on a sequence of specific parameters.

[0023] According to one aspect, as already described above, the test signal can be a step function, with the reward increasing the faster the step response occurs. The test signal can also be, for example, a sine signal, with the reward increasing the more accurately the sine function is represented.

[0024] Another aspect concerns a software agent of a reinforcement learning system that was trained using the method described above. A further aspect then concerns a method for parameterizing an actuator using the software agent. Thus, the following method is intended to perform parameterization, i.e., an optimal selection of control parameters for an unknown actuator. This means that the optimal parameterization is to be determined individually for an actuator, for example, at the end of a production line, taking manufacturing tolerances, etc., into account. Thus, the method discussed below is used to individually parameterize each produced actuator.

[0025] To do this, one of the aforementioned test signals and an initial parameterization are first entered into the actuator. This is done using the actuator itself, which is to be parameterized.

[0026] The reaction of this actor to the input is then determined and the corresponding parameters recorded. These are in turn entered into the software agent of the reinforcement learning system. The agent then begins to optimize the parameterization with the knowledge acquired during training, based on the reaction of the system and thus the reaction of the environment, in such a way that a reward for the software agent is optimized. The software agent first generates an initial parameterization, which is then entered into the actor. The actor, in turn, reacts with a corresponding state defined by the parameters, which are then entered into the software agent. This process is repeated until a corresponding termination condition is reached. This termination condition can, for example, be determined by a number of iterations to be performed.However, a termination condition can also be defined by achieving a certain behavior of the actuator, such as a behavior close to an optimal parameterization of the actuator.

[0027] The method thus allows for the parameterization of an actuator, for example, at the end of an actuator production line, and for individual parameterization. This parameterization can then be stored, for example, in a memory. The parameterization with the highest reward is preferably stored. To associate the parameterization with the actuator and thus assign the parameterization to the actuator, the parameterization can, for example, be stored in a memory with a corresponding serial number or a unique identifier of the actuator.

[0028] A further aspect relates to a device for carrying out the method according to one of the preceding claims.

[0029] Furthermore, a further aspect relates to an actuator which has been parameterized using the preceding method.

[0030] Short description of the characters

[0031] Figure 1 shows a flowchart of a procedure for training the software agent.

[0032] Figure 2 shows a flowchart of a method for parameterizing an actuator.

[0033] Detailed description of embodiments

[0034] Figure 1 first shows a flowchart of a method for training a software agent of a reinforcement learning system for the optimized control of an actuator.

[0035] In step S11, a test signal and an initial parameterization are first entered into an actuator representation of the actuator. In the present example, the test signal is a step function. With optimal parameterization, a step response from the actuator should therefore occur in the shortest possible time. The best parameterization and the path to this is to be learned by the software agent. For this purpose, the actuator's response to the entered parameterization and the entered test signal is determined in step S12. For this purpose, parameters that represent the actuator's response are derived and measured or determined. The actuator's behavior in the form of the parameters is then fed back to the software agent in step S13. Depending on how close a corresponding behavior of the actuator is to the desired behavior, the software agent receives a reward or a punishment.It then outputs parameters again, which are received by the software agent in step S14 and input to the actuator in step S15. The response to this input is then determined by the actuator in step S12, and the subsequent steps are repeated. Based on the interaction with the environment, i.e., with the actuator, the software agent can thus learn the behavior of the actuator depending on the change in the parameters and improves the parameters by striving for a higher reward and thus an improvement in the behavior of the actuator and thus towards a desired behavior.

[0036] This training can then be performed with various actuators. The result of this process, shown in Figure 1, is a software agent that can be used to parameterize actuators unknown to the system.

[0037] For this purpose, the method shown in Figure 2 is used. First, in step S21, a test signal and an initial parameterization are input to the unknown actuator to be parameterized, and the corresponding reaction of the actuator is determined in the form of parameters in step S22. These parameters are then input to the software agent in step S23, which, using its previously learned behavior, generates new parameterizations, which are received by the software agent in step S24. These new parameterizations are then input to the actuator again in step S25. A corresponding reaction is then determined again in step S22. This method is repeated until a termination condition is reached, for example, until 50 runs of this method have been performed.As an additional or alternative termination condition, a specific parameterization quality, i.e., a specific optimization of the parameterization, can also be specified. The optimal parameterization is then stored in a memory in step S22 along with a unique identifier of the actuator. Accordingly, the actuator can be delivered with the corresponding parameterization. This parameterization can then be stored in a control unit, and the actuator can be controlled using this parameterization.

[0038] This allows individual parameterization of all actuators without manual adjustment of the parameters.

[0039] Reference symbols Input of a test signal Determination of parameters Input of the parameters into the software agent Receiving a parameterization Input of the parameterization into the actor representation Input of a test signal Determination of parameters Input of the parameters into the software agent Receiving a parameterization Input of the parameterization into the actor Saving the parameterization with the highest reward

Claims

Patent claims 1 . A method for training a software agent of a reinforcement learning system for the optimized control of an actuator, comprising the steps: - inputting (S11) a test signal and an initial parameterization into an actuator representation of the actuator; - Determining (S12) parameters as a reaction of the actuator representation to the input; - inputting (S13) the parameters into the software agent of the reinforcement learning system, wherein the software agent is programmed such that a desired behavior of the parameters of the actuator for the test signal leads to an increased reward; - receiving (S14) a parameterization as output of the software agent; - entering (S15) the parameterization received as output of the software agent into the actor representation; and - Repeatedly performing the step of determining (S12) parameters, inputting (S13) the parameters into the software agent, receiving (S14) a parameterization as output of the software agent and inputting (S15) the parameterization received as output of the software agent until a termination condition is reached.

2. The method according to claim 1, wherein the actuator representation is either the actuator itself or a simulation of the actuator.

3. The method according to claim 1, wherein the steps of claim 1 are performed with different actuators and associated actuator representations.

4. The method according to claim 1, wherein the parameterization is at least one of control parameters for a current controller for controlling the actuator and control parameters of the actuator.

5. The method according to claim 1, wherein the test signal is one of a step function and a sine signal, the reward being increased the faster a step response occurs or the more accurately the sine function is mapped.

6. A software agent of a reinforcement learning system trained by the method of claim 1.

7. A method for parameterizing an actuator using the software agent according to claim 6, comprising the steps of: - Inputting (S21) a test signal and an initial parameterization into the actuator; - Determining (S22) parameters as a reaction of the actuator to the input; - entering (S23) the parameters into the software agent of the reinforcement learning system; - receiving (S24) a parameterization as output of the software agent; - inputting (S25) the parameterization received as output of the software agent into the actuator; and - Repeatedly performing the step of determining (S22) parameters, inputting (S23) the parameters into the software agent, receiving (S24) a parameterization as output of the software agent and inputting (S25) the parameterization received as output of the software agent until a termination condition is reached.

8. The method of claim 7, further comprising: - Saving (S26) the parameterization with the highest reward assigned to the actuator.

9. Apparatus for carrying out the method according to one of the preceding claims.

10. Actuator which has been parameterized using the method according to claim 7.

Images