Method, device and computer program for determining the strategy of an intelligent agent

Abstract

The present invention discloses a method, a device, a computer program and a machine-readable storage element. The intelligent agent implements actions according to strategies depending on the specific situation, so that the intelligent agent can autonomously achieve preset goals with the help of the implemented actions. The method includes the following steps. First, the environment of the intelligent agent is explored. Then, a model of the intelligent agent is created based on the exploration. The model has discrete state variables and continuous state variables, which are used to determine estimated sensor variables. In another step, a strategy is determined based on the created model. The present invention also relates to a computer program, a device for implementing the method, and a machine-readable storage element, on which the computer program is stored.

Description

Technical Field

[0001] The invention relates to a method for determining a strategy of an agent with the aid of a model of the agent's environment. The invention also relates to a device and a computer program, both of which are suitable for implementing the method. Background Art

[0002] It is known that an agent autonomously learns a policy to achieve a predefined task with the help of reinforcement learning. The agent explores its environment and thereby acquires knowledge about it. Based on this knowledge, the agent can plan actions to achieve the predefined task, depending on the situation. The disadvantage in this case is that a lot of data needs to be acquired through exploration in order to build up reliable knowledge about the environment.

[0003] DE 10 2017 218 811.1, which has not been published in advance, discloses a method for operating an actuator control system, which is suitable for adjusting a control variable of an actuator to a preset target variable. The actuator control system is also suitable for generating a manipulated variable according to a variable that characterizes the control strategy, and controlling the actuator according to the manipulated variable.

[0004] DE 10 2017 218 813.8, which has not been published in advance, discloses a method for automatically setting at least one parameter of an actuator control system. The actuator control system is mainly suitable for generating a manipulated variable based on at least one parameter, a target variable and a control variable, and controlling the actuator based on this manipulated variable, wherein a new value of at least one parameter is determined based on a steady-state probability distribution of the control variable. Summary of the invention

[0005] In a first aspect, the invention proposes a method for determining a strategy of an agent, in particular a robot. The agent is capable of performing actions according to the strategy, depending on the specific situation, so that the agent can autonomously reach a predefined goal with the help of these performed actions. The method mainly comprises the following steps: In a first step, the environment of the agent is explored according to the predefined actions performed by the agent. During the exploration, after the predefined actions performed by the agent, the environment is detected with the help of at least one sensor and provided as sensor variables. Optionally, these sensor variables correspond to each of the predefined actions performed. The parameterization of a model of the environment is configured based on the sensor variables and the predefined actions performed. The parameterization of the model is configured in such a way that the estimated sensor variables determined by the model according to the predefined actions performed are adapted to the provided sensor variables. The model is suitable for determining discrete state variables of the environment according to the predefined actions performed by the agent and in particular the sensor variables, and determining the estimated sensor variables according to the discrete state variables and in particular the parameters of the continuous state variables. The strategy is determined according to at least one sequence of possible actions and at least one reward, wherein the possible actions are determined with the help of the model.

[0006] “As the case may be” is to be understood as “depending on the environment detected by means of the sensor”. “Adapted” is to be understood as meaning that during the configuration of the environment model, the estimated sensor variables determined by the model are similar to the provided sensor variables, taking into account the cardinality of the model and the complexity of finding suitable parameterizations of the model.

[0007] The advantage of the method is that the model only needs to learn the association between the action and the discrete state generated thereby, rather than the association between the action and the resulting impact on the detected environment. Targeted knowledge about the association between the action and the environment can be learned thereby. Another advantage is that the sensor variables required to be detected in order to create the model are all reduced, because the association between the action and the discretized environment can be determined faster than the association between the action and the continuous environment, because the possible impact of the action on the continuous environment is much greater. This also affects the amount of calculation, because the amount of calculation is proportional to the sensor variable. Therefore, with the help of discrete state variables and in particular parameters based on continuous state variables, the cost for creating the model can be reduced, thereby also shortening the training duration.

[0008] In addition, complex high-dimensional problems involving unknown dynamic characteristics of the environment can be more specifically reflected through discrete state variables, thereby creating a more accurate model of the environment.

[0009] Furthermore, by measuring discrete state variables based on actions during model execution, the impact of the agent's actions on the environment can be measured in a resource-efficient manner.

[0010] The present invention further proposes that the model determines the reward based on discrete state variables. Preferably, in the process of determining the strategy, the arrival of a pre-set target is associated with a large reward.

[0011] Rewards are difficult to determine because they usually have a highly nonlinear relationship with the actions of the agent. The advantage of the method is that schemes for determining rewards based on discrete state variables are easier to learn. In addition, so-called Q-learning can also be used, for example.

[0012] The present invention further proposes that the continuous state variable also represents the prior knowledge about the environment.

[0013] The invention further proposes that the continuous state variable has information or additional knowledge about the environment, which can be used to determine, from the discrete state variable, in particular continuous and higher-dimensional estimated sensor variables, such as additional variables that clarify the discrete state variable. In addition or as an alternative, the continuous state variable can characterize the influence of the discrete state variable on the estimated sensor variable, or characterize a hidden state of the environment.

[0014] The present invention also proposes that the model is composed of an autoencoder, in particular a variational autoencoder, or a complex of several autoencoders.

[0015] The invention further proposes that the agent explores the environment according to the strategy and according to the environment detected by means of sensors, wherein at least the created model is adapted according to the provided sensor variables.

[0016] Additionally or alternatively, the strategy may be used to determine a control variable of an agent or an actuator, wherein the agent may be an actuator. The actuator may be, for example, an at least partially autonomous machine, an at least partially autonomous vehicle, a robot, a tool, a machine tool, or an aerial object such as a drone.

[0017] In another aspect, the present invention provides a computer program. The computer program is suitable for implementing one of the above methods. The computer program includes several instructions, and when the computer program is run on a computer, the instructions cause the computer to implement one of the methods including all the steps. The present invention also provides a machine-readable storage module, on which the computer program is stored. The present invention also provides a device, which is suitable for implementing one of the methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings illustrate embodiments of the above aspects, and these embodiments are described in detail below.

[0019] Figure 1 is a schematic diagram of an intelligent agent system capable of interacting with the environment;

[0020] Figure 2 is a schematic diagram of a first structure of a probabilistic graphical model;

[0021] Figure 3 is a schematic diagram of a second structure of a probabilistic graphical model;

[0022] Figure 4 is a schematic diagram of an at least partially autonomous vehicle including an intelligent agent system;

[0023] Figure 5 A schematic diagram of an embodiment of a method for determining a strategy of an intelligent agent system;

[0024] Figure 6 Schematic diagram of an apparatus for training a probabilistic graphical model. DETAILED DESCRIPTION

[0025] Figure 1 Schematic diagram of an intelligent agent system 10. The intelligent agent system 10 is adapted to autonomously learn a policy by exploring, preferably interacting with, an environment 11. Based on the policy and detected sensor variables (x), a decision module 12 determines an optimal action (a).

[0026] In another embodiment of the intelligent agent system 10, the decision module 12 has a probabilistic graphical model (PGM), which preferably models a Markov decision process (MDP). In this embodiment, the intelligent agent system (10) explores the environment 11 and creates a probabilistic graphical model (PGM) based on the exploration. The PGM models the environment 11 and the impact of the action (a) on the environment 11. Based on this, the intelligent agent system 10 can determine a strategy based on the PGM to determine an appropriate action (a) for a given sensor variable (x). The PGM can be configured by a number of parameters (P) stored in the memory 13.

[0027] The action (a) determined by the decision module 12 can be transmitted to the actuator (A). Subsequently, the actuator (A) implements interaction with the environment 11 according to the determined action (a). The interaction can be used to explore the environment 11 or to achieve a preset task.

[0028] The intelligent agent system 10 further comprises a computing unit 14 and a machine-readable storage element 15. On the storage element 15 a computer program comprising instructions may be stored which, when executed on the computing unit 14, cause the intelligent agent system 10 to operate by means of the computing unit 14.

[0029] Figure 2 Schematic diagram of a first time rollout 20 of a PGM calculation.

[0030] In this embodiment, the PGM is suitable for describing the environment 11 of the intelligent agent system 10, in particular, the impact and / or changes that occur to the environment 11 due to the action (a). The PGM can describe both the static characteristics (e.g., layout) of the environment 11 and the dynamic characteristics, in particular, the reaction of the environment 11 to the action (a) of the intelligent agent system 10 and / or the object, and preferably also describes the changes in the environment after the intelligent agent system 10 interacts with the environment 11.

[0031] In order to enable the PGM to describe the environment 11 as accurately as possible and with the least amount of computation, at least two auxiliary variables are introduced, namely a discrete state variable (B) and a continuous state variable (z). The discrete state variable (B) is used to characterize the environment 11 with the aid of a low-dimensional representation, such as several categories. The continuous state variable (z) is required to convert the information of the discrete state variable (B) in such a way that a detailed, preferably continuous and multidimensional representation (s) of the environment 11 is determined based on the discrete state variables, which can correspond to the environment measured with the aid of the sensor (S). In an ideal case, the representation (s) is an estimated sensor variable that the sensor (S) should detect based on the implemented action (a).

[0032] The PGM measures the discrete state variable (B) at any time point t. t ). Figure 2 In the t-1 , B t , B t+1 ) and the circles with time indexes t-1, t, t+1 are shown. t ) and according to the continuous state variable (z t ) Determine the measurable presentation (s t ). Then, according to the measurable presentation (s t )Determine the action(a) t ). Then determine a new discrete state variable (B t+1 ), which in turn generates a new measurable presentation (s t+1 ).

[0033] For example, PGM measures the discrete state variable (B t ), especially based on action (a t-1 ) and measurable presentation (S t-1 ) or the detected sensor variable (x).

[0034] Figure 2 By selecting from the t-1 ) to the circle with label (a t-1 ) is shown by the arrow of the circle. According to the discrete state variable (B t ) and according to the continuous state variable (z t ), measuring the measurable performance of the environment 11 (s t ). Figure 2 As shown, according to the discrete state variable (B t ) determines the continuous state variable (z t ). Measurable presentation(s) t ) may be an estimated sensor variable of the environment 11, wherein the sensor variable is the sensor in action (a) at time point t. t-1 ) The sensor variables that should be detected after implementation.

[0035] According to the measurable presentation (s t ), PGM can measure action (a t ). Figure 2 By selecting from the t ) to the circle with label (a t ) is shown by the arrow of the circle. t ), the PGM can then determine the best action (a t ) determines the subsequent discrete state variable (B) at time point t+1 t+1 ).

[0036] In this way, the sequence of actions and the reaction of the environment to the actions can be determined with the help of the PGM, in that the actions are determined in each case according to a measurable presentation (s). From this, a discrete state variable (B) is determined, from which a measurable presentation can in turn be determined. This makes it possible to plan several actions and preferably also to assess whether the actions lead to the desired result.

[0037] Preferably, the PGM is based on a discrete state variable (B t ) determines the reward. t ) can be used to determine strategies to determine measurable presentation (S t )'s best action (a tFor example, you can use rewards ( r t) Determine the strategy by maximizing the sum of the rewards for the actions or the total reward.

[0038] In PGM Figure 2 In another embodiment, shown by arrows between discrete states, the discrete state variable (B t-1 ) determines the discrete state variables.

[0039] Figure 3 Schematic diagram of the calculation of the PGM, in particular of a second time progression (30). The second time progression (30) differs from the first time progression (20) in that a continuous state variable (z) is not determined from a discrete state variable (B).

[0040] The following is an illustrative example of how PGM works, and Figure 2 and Figure 3 The possible meanings of the continuous state variable (z) and the discrete state variable (B) are described in detail. The intelligent agent system 10 needs to be applied in a vehicle, and the task of the vehicle is to leave the highway. To this end, the intelligent agent system 10 first explores the environment 11. Then a PGM is created. Among them, the discrete state variable (B) can, for example, characterize the speed state (slow, medium, high speed) and / or lane (right, left, middle) and / or distance (parallel to the exit, not far from the exit, far from the exit) of the vehicle. The continuous state variable (z) can, for example, specify the discrete state variable (B) more accurately and / or add other features, such as the potential speed distribution (p(v|v_diskret=slow)) based on the discrete state variable (B), and / or the distance distribution within a distance class (for example: p(d|e=parallel to the exit)). As a supplement or alternative, the continuous state variable (z) can characterize the distance distribution p(d_back|v=middle, s=left, e=not far from the exit) with adjacent vehicles. The measurable representation (s) may be an observable continuous variable p(s|z, B). The measurable representation (s) may be, for example, an image from a camera of the vehicle. With the aid of this PGM, a strategy may be determined so that the vehicle leaves the highway according to an action (e.g. lane change, deceleration).

[0041] Figure 4 Schematic diagram of an at least partially autonomous robot, which in a first embodiment is an at least partially autonomous vehicle (40). In another embodiment, the at least partially autonomous robot can be an at least partially autonomous robot, a service, installation or static production robot, or an autonomous flying object, such as a drone.

[0042] The at least partially autonomous vehicle 40 has a camera 41 acting as a sensor (S) which detects the environment 11 of the vehicle 40. In addition or as an alternative, a radar may be used as the sensor (S). The camera 41 is connected to the intelligent agent system 10. The intelligent agent system 10 determines an action (a) based on the image provided by the camera 41. This action (a) is transmitted to the actuator (A), which in turn Figure 4 In the middle is the control unit 43.

[0043] According to action (a), control unit 43 controls, for example, a motor or a brake system of vehicle 40 in such a way that vehicle 40 implements a strategy such as a lane change.

[0044] As described above, the task of the vehicle may be, for example, to exit a highway. Accordingly, the intelligent agent system 10 controls the vehicle to detect the environment 11, which in this case is a highway. After the intelligent agent system 10 understands its environment 11, the agent system can determine actions to control the vehicle to an exit and exit the highway through this exit. After the intelligent agent system 11 knows how to act, the vehicle can use this knowledge the next time it exits the highway and perform the required actions based on the detected images.

[0045] In another embodiment of the at least partially autonomous robot, the image of the camera 41 is transmitted to a server. The intelligent agent system 10 is executed on the server and the determined optimal action (a) is then transmitted back and provided to the control unit 43.

[0046] In another embodiment, the at least partially autonomous robot may be a tool, a machine tool or a manufacturing robot. Depending on the desired application, the intelligent agent system 10 is able to determine an action (a), for example, to process a workpiece in an optimal manner. The actuator may be, for example, a motor that drives a grinding head.

[0047] In an alternative embodiment, the intelligent agent system 10 that can be connected to a temperature sensor in this embodiment can be used for building control. In this case, the intelligent agent system 10 can be adapted to control an air conditioning or heating system based on the temperature sensor.

[0048] The intelligent agent system can also be used in the field of medical technology. For example, the intelligent agent system 10 can recommend the optimal dosage of a drug or the optimal treatment / therapeutic method based on the detected vital functions of the patient. For example, an image of a human or animal body or a part thereof can be detected and provided to the intelligent agent system 10. This can be achieved, for example, with the aid of optical signals, with the aid of ultrasonic signals, or with the aid of MRT / CT methods. As an alternative or supplement, measured vital functions such as pulse or body temperature can be provided to the intelligent agent system 10. As a supplement or alternative, for the intelligent agent system 10, the doctor can provide variables representing patient characteristics as input quantities to the intelligent agent system 10 by manual input. Preferably, the PGM presents the effects of therapeutic measures on the patient here.

[0049] Figure 5 is a schematic diagram of a method 50 for determining a strategy and illustrates optional downstream steps for operating the intelligent agent system 10 .

[0050] The method starts at step 51. In step 51, an exploration of the environment 11 is carried out. The exploration is carried out either randomly or according to a priori knowledge or according to a sequence of predefinable actions (each corresponding to a time point t in the time point sequence T). The actions and the sensor variables subsequently detected are preferably stored and provided as training data. The reward at each time point t is preferably also determined or measured and stored as a supplement to the training data.

[0051] Then step 52 is performed, in which the parameterization of the PGM is configured according to the training data provided from step 51. The PGM is configured such that it correctly predicts the response of the environment 11 to the action (a), in particular such that the PGM correctly determines the sensor variables of the training data according to said action.

[0052] Before configuring the parameterization, the structure of the PGM can be selected in step 52. The PGM may have Figure 2 or Figure 3 As an alternative, the continuous state variable (z) can be independent of or related to the discrete state variable (B). Figure 2 and Figure 3 Two PGMs are created. It should be noted that after the two PGMs are created, one of the two PGMs may be selected according to a pre-settable criterion (see step 52 “ELBO”).

[0053] According to the choice of the structure of the PGM, the following equations can be imparted to the PGM for determining the corresponding variables (B, z, s), in particular for determining the appropriate configuration of the parameters of the PGM. Figure 2For the structure shown, it is preferred to use Evidence Lower Bound (ELBO).

[0054]

[0055] where p θ is a probability distribution and presents Figure 2 The arrow in , and parameterized by θ, and, is a probability distribution that in particular behaves like Figure 2 The arrows shown are in the reverse direction. The advantageous effect of the reverse direction arrows is that the PGM can thereby be created more quickly.

[0056] For PGM Figure 3 For the structure shown, to create the PGM, the Evidence Lower Bound (ELBO) was also used with a slight modification.

[0057]

[0058] If in step 52, respectively Figure 2 and Figure 3 If two PGMs are created, then which of these PGMs is more suitable can be compared based on ELBO, and the PGM with a larger ELBO is preferred.

[0059] After step 52 is completed, step 53 follows, in which a strategy is determined according to the created PGM. To this end, at least one sequence of possible actions and the corresponding estimated sensor variables are determined by the PGM.

[0060] In step 53 , a strategy can be determined based on the sequence of possible actions by means of a so-called “value iteration”.

[0061] In a subsequent optional step 54, the environment 11 is re-explored using the strategy determined according to step 53. In addition, another strategy may be used for exploration. As in step 51, the exploration of the environment 11 may be recorded. The created PGM may then be adjusted, for example, by step 52, based on the recorded re-implemented exploration, so that the PGM more accurately reflects the environment 11.

[0062] In the event that no adjustments to the PGM need to be made, for example because the PGM reflects the environment sufficiently accurately in terms of possible actions, a final strategy is determined, which is then used, for example, by the decision module 12 to achieve a predeterminable task.

[0063] After the method 50 is terminated at step 54 , the method 50 may be restarted at step 52 , for example in a loop, if the PGM is found to be too inaccurate or if the PGM needs to be adjusted for another application and / or task.

[0064] The exploration by means of the intelligent agent system 10, for example by means of the vehicle 40, can also be recorded and transmitted to a server, which then performs steps 52, 53 and, if appropriate, part of step 54. The server sends back the results, for example a trained PGM or strategy.

[0065] After the method 50 is executed, the intelligent agent system (10) is optionally operated with the aid of the strategy, for example, the vehicle 40 is controlled according to the action.

[0066] Figure 6 Schematic diagram of a device 60 for creating a PGM, in particular for implementing step 52 of method 50. The device 60 comprises a training module 61 and a decision module 12. The device 60 trains the PGM according to the provided training data. The training data preferably comprises a number of acquired images and corresponding implemented actions, which preferably correspond to a reward each. During the training, the parameters (P) of the PGM stored in the memory 13 are adjusted.

Claims

1. A method for determining a strategy of an intelligent agent system, wherein the intelligent agent system determines an action (a) for a given sensor variable (x) according to a probabilistic graphical model (PGM), wherein the intelligent agent system is applied in a vehicle, characterized in that: According to the action (a t-1 , a t ) and at least one reward (r t ) determining said strategy, The action (a) is determined by means of a parameterized probabilistic graphical model PGM of the intelligent agent system t-1 , a t , a t+1 ), The probabilistic graphical model PGM is suitable for determining discrete state variables (B) of the environment according to preset actions implemented by the intelligent agent system and sensor variables. t ), which characterizes at least one state of the environment and / or the intelligent agent system, and for each of several preset categories, indicates whether the at least one state belongs to the corresponding category, the discrete state variable (B t ) is used to characterize the speed state and / or lane and / or distance of the vehicle, The probabilistic graphical model PGM is also suitable for determining discrete state variables (B) of the environment according to preset actions implemented by the intelligent agent system and sensor variables. t ) determines the estimated sensor variable (S t ), including the following steps: Explore the environment according to the preset actions implemented by the intelligent agent system, wherein during the exploration process, after the implementation of a preset action by the intelligent agent system, the environment (11) is detected by means of at least one sensor and provided as a sensor variable (x); configuring the parameterization of the probabilistic graphical model PGM according to the implemented predefined actions and the provided sensor variables (x), The parameterization of the probabilistic graphical model PGM is configured in such a way that the probabilistic graphical model PGM determines the estimated sensor variables (s t ) is adapted to the provided sensor variable (x); the probability graphical model PGM is also suitable for the discrete state variable (B t ) and according to the continuous state variable (z t ) parameters to determine the estimated sensor variable (s t ), the continuous state variable (z t ) is a representation of the spacing distribution with adjacent vehicles, The continuous state variable (z t ) parameters for the continuous state variable (z t ), wherein the probability distribution characterizes at least one state of the environment and / or the intelligent agent system; With the help of the discrete state variable (B t ) is compared to the representation of the state of the environment and / or intelligent agent system through discrete state variables (B t ) represents the state, the continuous state variable (z t ) to achieve a more accurate representation of the state and determine a strategy to control vehicle driving with the help of the probabilistic graphical model PGM.

2. The method according to claim 1, characterized in that The probabilistic graphical model PGM is also suitable for determining the discrete state variables (B t ) determines the reward (r t ).

3. The method according to claim 2, characterized in that In the process of configuring the parameterization, the parameterization is configured such that the probabilistic graphical model PGM is configured according to the discrete state variables (B t ) and the preset rated reward determination reward (r t ), wherein the preset rated reward corresponds to the implementation of the corresponding discrete state variable (B t ) actions.

4. The method according to claim 1, characterized in that According to the discrete state variable (B t ) determines the continuous state variable (z t ) parameters.

5. The method according to claim 4, characterized in that The discrete state variable (B t ) comprises a plurality of values, wherein each of the plurality of values ​​characterizes a probability for exactly one of the plurality of preset categories, indicating that the at least one state belongs to this category.

6. The method according to claim 1, characterized in that The intelligent agent system re-explores the environment according to the strategy and according to the environment detected by the sensor, Wherein the probabilistic graphical model PGM is reconfigured at least according to the provided sensor variables.

7. The method according to claim 1, characterized in that An action is determined according to the strategy, wherein a control variable is determined according to the action for controlling a physical actuator by means of a control unit (43).

8. The method according to claim 7, characterized in that An at least partially autonomous robot or a manufacturing system or a personal assistant system or an access system or a monitoring system or a medical imaging system is controlled in dependence on the control variables.

9. A device for determining the strategy of an intelligent agent system, characterized in that: Suitable for implementing the method according to any one of claims 1 to 8.

Images