Apparatus and method for controlling an agent

Abstract

Methods for controlling an agent are described by various embodiments. [Solution] The method includes: determining a control history that shows a series of actions performed by the agent leading up to the current state and observations of changes in the agent's state and / or the state of the agent's environment, with respect to the agent's current state and the state of the environment the agent is to control; determining the encoding of the control history by supplying the control history to a history encoder including a Kalman filter, the encoding being given by a system state estimate determined by the Kalman filter; supplying the encoding to a control policy trained to determine actions from the control policy encoding; and controlling the agent to perform the actions provided by the control policy in response to the supply of the encoding.

Description

[Technical Field]

[0001] This disclosure relates to an apparatus and method for controlling an agent. [Background technology]

[0002] Reinforcement learning (RL) is a machine learning paradigm that enables machines to learn to perform predetermined behaviors related to task specifications, such as controlling actions to reach a target location in a robot navigation scenario. Learning policies that generate these behaviors using reinforcement learning differs from learning using supervised learning in terms of how training data is constructed and acquired. In supervised learning, the provided training data consists of matched pairs of inputs to the policy (e.g., observations such as sensory readings) and predetermined outputs (actions to be taken), whereas in reinforcement learning, there is no fixed training data provided. Policies are learned from experiential data (i.e., observations) collected through the interaction between the machine and the environment. This provides the machine with feedback (reward) signals that score / evaluate actions taken in a given context (state). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] The paper, “Temporal Parallelization of Bayesian Smoothers”, by Simo Sarkka and Angel F. Garcia-Fernandez et al., is published in IEEE Transactions on Automatic Control, 66(1):299 - 306, January 2021. [Non-Patent Document 2] The paper, "On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learning," by P. Becker et al., is published in Transactions on Machine Learning Research on October 10, 2022. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] In actor-critic models, the determination of the next course of action (i.e., policy), and the estimation of the value of the action taken (or state achieved), may be based not only on recent observations and actions but also on previous observations and actions (i.e., historical data) to enable better control. However, this complicates the input to the corresponding modules (e.g., actors (i.e., policies) and critics). Therefore, efforts are needed to enable efficient information input from historical data to policies and / or critics.

[0005] The paper “Temporal Parallelization of Bayesian Smoothers”, IEEE Transactions on Automatic Control, 66(1):299 - 306, January 2021 by Simo Sarkka and Angel F. Garcia-Fernandez et al. (hereinafter referred to as reference [1]) describes a temporal parallelization algorithm for Bayesian smoothers, particularly Kalman filters.

[0006] The paper “On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learning”, Transactions on Machine Learning Research, October 10, 2022 (hereinafter referred to as reference [2]) by P. Becker et al. describes Kalman filtering using structured state space models. [Means for solving the problem]

[0007] Various embodiments provide a method for controlling an agent, which includes determining a control history that shows a series of actions performed by the agent leading up to the current state and observations of changes in the agent's state and / or the agent's environment state, with respect to the agent's current state and the state of the environment in which the agent is to be controlled; determining the encoding of the control history by supplying the control history to a history encoder including a Kalman filter, the encoding being given by a system state estimate determined by the Kalman filter; supplying the encoding to a control policy trained to determine actions from the control policy encoding; and controlling the agent to perform actions provided by the control policy in response to the supply of the encoding.

[0008] The above method enables the compression of historical data while solving problems that require inference regarding uncertainty by using internal probabilistic filtering (in the form of a Kalman filter). For example, when a control target system (e.g., an agent in an environment) outputs noisy observations, the Kalman filter can output a filtered latent representation, which can then be used for policy optimization. Also, the Kalman filter may be stacked, and more complex architectures are possible. The Kalman filter may have trainable parameters that can be end-to-end trained with the policy. This means that the representation of uncertainty used for filtering the latent state is learned by a method aimed at maximizing the return of the policy.

[0009] The above method may be applied in the context of reinforcement learning under partial observability, where the reinforcement learning (RL) model cannot access the underlying state of the system to be controlled but can infer such a state from the history of past observations and actions (i.e., control history). For example, a system with noisy observations or a system whose parameters change over time fits this setting. The method may be used for online training of the policy under such conditions. The method internally implements probabilistic filtering for a linear system (in the form of a Kalman filter), and its parameters can be directly trained through an RL loss function aimed at maximizing the expected return. This probabilistic filtering functions as an inductive bias for learning a good latent representation for control.

[0010] Examples are shown below.

[0011] Example 1 is a method for controlling an agent as described above.

[0012] Example 2 is the method according to Example 1, which includes training a control policy, wherein the parameters of the Kalman filter are trained together with the control policy.

[0013] For example, the entire control architecture (i.e., the control pipeline including the Kalman filter, i.e., the history encoder, and the policy) may be trained end-to-end. By training the Kalman filter together with the control policy, it is ensured that in the generation of the encoding by the Kalman filter, the information necessary for the effective selection of control actions is maintained (i.e., not lost in the encoding).

[0014] Example 3 is the method according to Example 1 or Example 2, which includes training a control policy using reinforcement learning.

[0015] Thereby, the control policy and the Kalman filter can be effectively trained together. In other words, various embodiments provide a method for training a control policy for controlling an agent, the method including: executing an action using the agent (selected by the control policy in response to the supply of the control history encoding); observing the state transition of the agent and / or the environment of the agent in response to the action; observing the reward received from the state transition; and training the agent using reinforcement learning according to the reward received from the state transition.

[0016] Example 4 is the method according to any one of Examples 1 to 3, wherein the Kalman filter is configured to estimate the system state using a linear structured state space model for the system state and observations given by a trainable matrix having a diagonal structure.

[0017] Thereby, stability is provided when processing a long sequence.

[0018] Example 5 is a method according to any one of Examples 1 to 4, which includes parallel processing of multiple control histories.

[0019] For example, the trajectory determined during rollout may be divided into sub-trajectories that are processed in parallel to speed up training.

[0020] Example 6 is a method of any one of Examples 1 to 5, wherein the Kalman filter is configured to repeat the system state estimate determined by the end of the sequence (in each estimation iteration, i.e., prediction and update iteration) for control history indicating that the sequence is shorter than the default length, until it reaches a number of estimation iterations corresponding to the default length.

[0021] This enables parallel processing of control histories of different lengths (e.g., orbits or sub-orbits of orbits). Initiating the Kalman filter to perform iterations of the last system state estimate can be achieved by using a masked binary operator, as described below (see equation (4)). This allows for parallel processing without the need to embed some sequences, which may not be clearly defined in non-discrete sequences, with arbitrary masking values.

[0022] Example 7 is a method of any one of Examples 1 to 6, wherein the method determines the encoding of the control history by feeding the control history to the first Kalman filter of a sequence of one or more Kalman filters, and feeding the system state estimates of each Kalman filter in the sequence to the next Kalman filters, excluding the last Kalman filter in the sequence, the encoding being given by the system state estimate (e.g., an equivalent value) determined by the last Kalman filter in the sequence.

[0023] In other words, to determine the encoding, multiple Kalman filters (i.e., Kalman filter layers, each implementing a Kalman filter) may be stacked (i.e., applied sequentially). This allows for greater flexibility in encoding. The last Kalman filter in the sequence (which may be a single Kalman filter) should be understood as corresponding to the Kalman filter described in Example 1. This Kalman filter may be connected via a linear layer to ensure consistency between the input and output dimensions.

[0024] Example 8 is a controller configured to perform the method described in any one of Examples 1 through 7.

[0025] Example 9 is a computer program that, when executed by a computer, includes instructions to cause the computer to perform a method described in any one of Examples 1 through 7.

[0026] Example 10 is a computer-readable medium that, when executed by a computer, contains instructions causing the computer to perform the method described in any one of Examples 1 through 7.

[0027] In the drawings, similar reference numerals generally indicate equivalent parts across different figures. These drawings are not necessarily to scale, and rather the emphasis is on illustrating the general principles of the invention. In the following description, various embodiments will be described with reference to the following drawings. [Brief explanation of the drawing]

[0028] [Figure 1] This figure shows a control scenario according to the embodiment. [Figure 2] This figure shows a recurrent actor-critic architecture as an example of a reinforcement learning architecture using a history encoder. [Figure 3]This figure shows a Kalman filter (KF) layer according to the embodiment. [Figure 4] This flowchart shows the methods for controlling the agent. [Modes for carrying out the invention]

[0029] The following detailed description refers to the accompanying drawings, which illustrate specific details and embodiments of the present disclosure for carrying out the invention. Other embodiments may be used without departing from the scope of the invention, and may involve structural, logical, and electrical modifications. The various embodiments of the present disclosure are not necessarily mutually exclusive, and some embodiments of the present disclosure may be combined with one or more other embodiments of the present disclosure to form new embodiments.

[0030] The following sections will provide more detailed explanations of various examples.

[0031] Figure 1 shows the control scenario.

[0032] The robot 100 is located within the environment 101. The robot 100 has a starting position 102 and should reach a target position 103. The environment 101 includes obstacles 104 that the robot 100 should avoid. For example, these obstacles 104 may not be passable by the robot 100 (e.g., they are walls, trees or rocks), or they may be avoided for reasons that could damage or injure the robot (e.g., pedestrians).

[0033] The robot 100 has a controller 105 (which may remotely control the robot 100, i.e., the robot 100 is controlled by remote control). In the exemplary scenario of Figure 1, the objective is for the controller 105 to control the robot 100 and guide it through the environment 101 from a starting position 102 to a target position 103. For example, the robot 100 is an autonomous vehicle, but may be a robot equipped with legs or tracks, or other types of propulsion systems (such as a deep-sea rover or a Mars rover).

[0034] Furthermore, the embodiment is not limited to a scenario in which the robot (as a whole) is to move between position 102 and position 103, and its end effector may be used to control a robot arm that is to move between position 102 and position 103 (without collision with obstacle 104).

[0035] Therefore, in the following, terms such as robot, vehicle, and machine are used as examples of “objects” to be controlled, i.e., as examples of computer-controlled systems (e.g., machines). The approaches described herein can be applied to different types of computer-controlled machines, such as robots and vehicles. In the following, the general term “robot device” is also used to refer to all types of technical systems that can be controlled by the approaches described below. The environment may be simulated, and for example, a control policy may be a control policy for a virtual vehicle or other movable device, for example, in a simulation to test other policies for autonomous driving.

[0036] Ideally, the controller 105 learns a control policy that enables it to successfully control the robot 101 (from the starting position 102 to the target position 103 without colliding with obstacles 104) for any scenario (i.e., environment, starting and target positions), especially for specific scenarios that the controller 105 has never encountered before.

[0037] Therefore, various embodiments relate to learning control policies for a particular task (or its distribution) through interaction with the environment 101. During training, the scenario (in particular the environment 101) may be simulated, but typically it will be realistic in deployment.

[0038] One approach to learning control policies is reinforcement learning (RL), where the robot 100 and / or its controller 105 act as reinforcement learning agents.

[0039] Reinforcement learning (RL) is a technique for learning control policies. An RL algorithm iteratively updates the parameter θ of a parametric policy πθ(a|s), represented, for example, by a neural network, which maps a state s (e.g., a (preprocessed) sensor signal) to an action a (a control signal). During training, the policy interacts with the (possibly simulated) environment 101 episodically (i.e., in one or more episodes) during the rollout. During the (possibly simulated) training rollout in environment 101, the controller 105, according to the current control policy, updates the current state s at each discrete-time step t. t The action a is performed according to the following, which leads to a new state s in the next discrete-time step. t+1 This leads to a reward r t This reward is received and tis used for policy updates. (Training) Rollout ends once the target state is reached, the accumulated (potentially discounted) reward exceeds a threshold, or the maximum number of time steps, the planning horizon T, is reached. During training, the objective function that depends on the reward (e.g., the discounted sum of rewards received during rollout) is maximized by updating the policy parameters. In the case of an actor-critic RL scheme such as the example below, training also includes updating the critic. Training ends once the policy meets a specific quality criterion with respect to the objective function, the maximum number of policy updates is executed, or the maximum number of steps is executed in the (simulation) environment.

[0040] In the following example, the state space S, action space A, observation space O, planning horizon

Number

Number

[0041] At time step t of an episode in M, the agent observes o t ~O(·|s t ), selects an action a

Number

[0042] In a typical setup, the (RL) agent includes the following: (i) A probabilistic policy π:H that maps observed history to a behavioral distribution. t →P(A) (the policy parameter θ is omitted here for simplicity) (ii) Based on the history and (current) behavior, the following is expected under the policy:

number

number

number

[0043] The weakness of the approach following the general formulation of RL in POMDP described above is that both the policy and the value function depend on the entire history, which becomes cumbersome in almost all cases except for very small ones. Instead, practical algorithms seek to compress the history into a compact representation.

[0044] One common framework for learning such representations is to pass them through a history encoder, which extracts some latent representation z from the observed history. t :=φ(h :t ) ∈ Z mapping φ:Ht →It can be defined by Z. Below, we will use notation somewhat arbitrarily, π(a t |z t ) and Q π (z t ,a t ) represents the policy and value under this latent representation, respectively.

[0045] Various embodiments utilize a history encoder based on a recurrent Kalman network (RKN) to implement simple probabilistic inference for latent states. In other words, a history encoder with one or more layers is used, each layer operating according to a Kalman filter.

[0046] The Kalman filter operates based on a linear dynamic system discretized in the time domain. In various embodiments, for this purpose, the following equation is used:

number

number

number

number

number

[0047] In practical terms, as is generally the case,

number

number

[0048] In order to introduce uncertainty into the state-space model, we follow equation (2).

number

number

number

number

[0049] Closed-form solutions exist for Kalman filtering using such models, and these may be used for implementing Kalman filters. However, these require matrix inverse operations, making them computationally expensive and sometimes unsuitable for gradient-based learning. Therefore, according to various embodiments, a simplified inference scheme consisting of simple element-wise addition and multiplication is used for Kalman filtering. In particular, diagonal structured SSMs are suitable for simple Kalman filtering equations, such as those given in reference [2]. One of the main advantages of using linear recursion and simplified inference schemes is that they can be efficiently implemented using parallel scans. For an input sequence of length K, the execution time complexity of a parallel scan is 0(log(K)), given enough parallel processors. The condition for parallel scans is to define the sequence processing problem in terms of the concatenation operator ●, so that (a●b)●c=a●(b●c) is maintained for any triplet (a,b,c) of elements. Linear SSMs and their associated stochastic filters have such properties (see reference [1]).

[0050] Figure 2 shows the recurrent actor-critic architecture 200 as an example of a reinforcement learning architecture using a history encoder.

[0051] Each of the actors 201 and the critic 202 includes embedders 203,204 that generate history from observations and actions (as described above). History encoders 205,206,207 encode the history on a latent state basis and use this as input for the policy. The policy is implemented by the first multilayer perceptron 208, and similarly, two versions of the value function are implemented by the second multilayer perceptron 209 and the third multilayer perceptron 210. The use of two value functions is merely an example here, and a single value function may also be used. The stability of training can be improved by using two value functions, for example, by using the minimum of their outputs as the value estimate. This architecture may be trained end-to-end according to various types of (standard) actor-critic reinforcement learning and various (actor-critic) loss functions, for example, by training with a Soft Actor Critic (SAC) loss aimed at maximizing the (soft) Q-score.

[0052] As mentioned above, each of the history encoders 205, 206, and 207 includes one or more Kalman filter layers.

[0053] Figure 3 shows a Kalman filter (KF) layer 300 according to one embodiment.

[0054] These multiple Kalman filter layers may be stacked together to constitute the history encoders 205, 206, and 207, for example, as well as non-stochastic SSM layers and their derivatives. In contrast to a standard SSM layer, the KF layer 300 filters the latent state

number

number

number

[0055] Therefore, the KF layer 300 is the history sequence h :t It receives and interprets it as three distinct signals in latent space: input (i.e., action) U :t , observed value W :t , and the observation noise (diagonal) covariance Σ w,:t These sequences are projected onto the sequence. These sequences are processed by the Kalman filter 305 according to the standard Kalman filtering equations. These equations are logarithmically scaled by the sequence length using parallel scanning. Finally, the history coding Z is applied. :t To obtain the posterior mean latent state

number

[0056] To improve computational efficiency during training, in various embodiments, architecture 200 (i.e., a controller, e.g., controller 105 implementing the architecture) generally handles batches of variable-sized orbits. On the other hand, efficient batch execution of parallel scans requires sequences of the same size (i.e., all sequences have a default length). This mismatch can be easily addressed in some sequence modeling tasks (such as languages) by introducing special masking tokens used to embed sequences up to a common maximum length. However, in common cases, it may not be easy to define a suitable mask value. In particular, if the data is not discrete, the choice of mask value becomes arbitrary.

[0057] Alternatively, the join operator may be modified to natively handle sequences of variable size. For example, in (particularly off-policy) RL, a subsequence (e.g., a sub-trajectory) of an episode (i.e., the complete trajectory derived from the episode) is a sample as training input, and the join operator handles identical states (i.e., in this application, latent state z) across embedded steps. t It is designed to embed shortened sequences by propagating such concatenation operators.

number

number

number

number

number

number

[0058] In summary, various embodiments provide a method as shown in Figure 4.

[0059] Figure 4 shows a flowchart 400 illustrating a method for controlling an agent (for example, a technical system such as a robotic device, e.g., a robot or a vehicle).

[0060] In 401, a control history is determined that shows the current state of the agent and the state of the environment in which the agent is to be controlled, as well as a series of actions performed by the agent leading up to the current state, and observations regarding changes in the state of the agent and / or the state of the agent's environment (caused by the series of actions).

[0061] In 402, the encoding of the control history is determined (generated) by supplying the control history to a history encoder including a Kalman filter (i.e., the input expected by the Kalman filter, i.e., a series of measurements observed over the time period expected by the Kalman filter, is given by the control history (or at least derived therefrom, e.g., by one or more preceding Kalman filters)), where the encoding is given by the system state estimate determined by the Kalman filter (directly from the control history, or from a (pre)processed version of the control history, e.g., by one or more preceding Kalman filters).

[0062] In a 403, the encoding is supplied to the control policy (or actor) that has been trained to determine its actions from the control policy encoding. This encoding is also supplied to the critic in cases where actor-critic RL is used.

[0063] In a 404 error, the agent is controlled to perform the actions provided by the control policy, depending on the encoding provided.

[0064] The approach in Figure 4 can be used to compute control signals for controlling technical systems such as computer-controlled machines, including robots, vehicles, home appliances, power tools, manufacturing machinery, personal assistants, or access control systems (here, the technical system, or the controller of the technical system, can be considered an agent that is sequentially controlled according to its control policy and therefore "controlled" by that control policy). In various embodiments, a policy for controlling the technical system may be learned, and the technical system may be operated in accordance with that policy.

[0065] Various embodiments can receive and use various types of sensor data to provide information about the state of the environment and agents (e.g., technical systems), that is, to collect observations in the form of one or more discrete or continuous signals. This includes all kinds of measurements (force, velocity, etc.), as well as image data (i.e., digital images) from various visual sensors (cameras) such as video, radar, LiDAR, ultrasound, thermal imaging, motion, and sonar.

[0066] The method in Figure 4 may be carried out by one or more data processing devices (e.g., a computer or microcontroller) having one or more data processing units. The term “data processing unit” should be understood to mean any kind of entity that enables the processing of data or signals. For example, data or signals may be processed according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit may include or can be composed of analog circuits, digital circuits, logic circuits, microprocessors, microcontrollers, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), or any combination thereof. It should be understood that any other means for carrying out each of the functions described in more detail herein also include data processing units or logic circuits. One or more of the method steps described in more detail herein may be carried out (e.g., implemented) by the data processing unit through one or more specific functions performed by the data processing unit.

[0067] Therefore, in one embodiment, this method is implemented in a computer.

Claims

1. A method for controlling agent (100), Determining a control history (401) that shows the current state of the agent (100) and the state of the environment (101) of the agent (100) that the agent (100) should control, which shows a series of actions performed by the agent (100) up to the current state and shows observations regarding changes in the state of the agent (100) and / or the state of the environment (101) of the agent (100), Determining the encoding of the control history by supplying the control history to a history encoder (205) including a Kalman filter (305), wherein the encoding is given by the system state estimate determined by the Kalman filter (305) (402), (403) supplying the coding to a control policy that has been trained to determine an action from the control policy coding, Controlling the agent (100) to perform actions provided by the control policy in response to the supply of the encoding (404), A method that includes this.

2. The method according to claim 1, comprising training the control policy, wherein the parameters of the Kalman filter are trained together with the control policy.

3. The method according to claim 1 or 2, comprising training the control policy using reinforcement learning.

4. The method according to any one of claims 1 to 3, wherein the Kalman filter (305) is configured to estimate the system state using the system state given by a trainable matrix having a diagonal structure and a linearly structured state space model for the observation.

5. The method according to any one of claims 1 to 4, comprising parallel processing of multiple control histories.

6. The method according to any one of claims 1 to 5, wherein the Kalman filter (305) is configured to repeat the system state estimate determined by the end of the sequence until a plurality of estimation iterations corresponding to the default length are reached, for control history indicating that the sequence is shorter than the default length.

7. The method according to any one of claims 1 to 6, wherein the encoding of the control history is determined by supplying the control history to the first Kalman filter (305) of a sequence of Kalman filters (305), and supplying the system state estimates of each Kalman filter (305) in the sequence to the next Kalman filter (305), excluding the last Kalman filter in the sequence, the encoding is given by the system state estimate determined by the last Kalman filter (305) in the sequence.

8. A controller configured to carry out the method described in any one of claims 1 to 7.

9. A computer program that, when executed by a computer, includes instructions causing the computer to perform the method described in any one of claims 1 to 7.

10. A computer-readable medium comprising, when executed by a computer, instructions causing the computer to carry out the method according to any one of claims 1 to 7.

Images