Devices and computer-implemented methods for continuous-time interaction modeling of agents

Abstract

Regarding a device and a computer-implemented method for continuous-time interaction modeling of agents, the method comprises: a step (202) of providing a latent state of a first agent and a latent state of a second agent, in particular, characterizing the position or velocity of these agents; a step (204) of providing a first Gaussian process distribution for a first function for modeling the kinematic behavior of an agent independently of other agents, and a second Gaussian process distribution for a second function for modeling the interaction between agents; a step (206) of sampling the first function from the first Gaussian process distribution and the second function from the second Gaussian process distribution, wherein the first function is configured to map the latent state of one agent to the contribution to the change in the latent state of that agent, and the second function is configured to map the latent states of two agents to the contribution to the change in the latent state of one of the two agents, and the method comprises a step (208) of changing the latent state of the first agent according to a first contribution resulting from the mapping from the latent state of the first agent to the first contribution by the first function and a second contribution resulting from the mapping from the latent states of the first agent and the second agent to the second contribution by the second function.

Description

Technical Field

[0001] The present invention relates to devices and computer-implemented methods for continuous-time interaction modeling of agents, particularly objects of interest.

Background Art

[0002] Learning the behavior of unknown dynamic systems from data is a fundamental problem in machine learning.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The computer-implemented method and device according to the independent claims are based on a Gaussian process (GP), operate in continuous time, and use a model that decomposes complex continuous dynamics into independent kinematic and interaction components to explain interpretable non-linear dynamics. This model treats the independent kinematics of the agents and their interactions separately. Function-level regularization via GP is the key to learning the disentangled representation. This model is based on ordinary differential equations (ODEs). This explains many complex dynamics that have a natural representation with respect to time differences. The continuous-time formulation enables the direct integration of domain knowledge that exploits the inductive bias of the model. Thus, the kinematics of the objects and their interactions are modeled separately with two different Gaussian processes.

Means for Solving the Problems

[0004] A computer-implemented method for continuous-time interaction modeling of agents includes the steps of: providing latent states of a first agent and a second agent, in particular characterizing the positions or velocities of these agents; providing a first Gaussian process distribution with respect to a first function for modeling the kinematic behavior of agents independently of other agents, and a second Gaussian process distribution with respect to a second function for modeling interactions between agents; and sampling a first function from the first Gaussian process distribution and a second function from the second Gaussian process distribution. The method includes a first function configured to map the latent state of one agent to a contribution to the change of that latent state, and a second function configured to map the latent states of two agents to a contribution to the change of the latent state of one of the two agents, and the method includes the step of changing the latent state of the first agent in accordance with a first contribution resulting from the mapping of the latent state of the first agent to a first contribution by the first function, and a second contribution resulting from the mapping of the latent state of the first agent and the latent state of the second agent to a second contribution by the second function.

[0005] The method preferably includes the steps of providing initial latent states for a plurality of agents, including a first agent and a second agent, and changing the latent state of the first agent according to a second contribution resulting from mapping pairs of latent states of the first agent and a different second agent among the plurality of agents using a second function, or selecting a subset of the plurality of agents and changing the latent state of the first agent according to a second contribution resulting from mapping pairs of latent states of the first agent and a different second agent of the subset using a second function. Thus, the method considers either all agents different from the first agent or the agent's neighborhood for interaction. Agents that are not in the agent's neighborhood are less likely to interact with the agent. This reduces the computational load while maintaining reasonable accuracy of the model.

[0006] This method preferably includes the step of determining agents from a group of agents with respect to a subset of agents that are closer to a first agent than to the other agents, according to a measure of distance between agents. Thus, this method considers the proximity of agents for interaction. The metric may be a measure of any kind of property of the agents. For agent movement, the distance from each other is a preferred metric.

[0007] The method preferably includes the step of providing a data sequence, wherein the step of providing the initial latent state of a first agent and / or a second agent includes determining the initial latent state using an encoder configured to map the data sequence to that initial latent state.

[0008] The method preferably includes the step of determining an output based on the latent state of a first agent, particularly a trajectory sample, preferably the trajectory of the first agent's position and / or velocity over time. The output can relate to any kind of characteristic of the agent. For the agent's movement, a trajectory sample is a preferred output.

[0009] Preferably, the first Gaussian process distribution includes a posterior distribution, and the method includes the steps of learning a particularly sparse approximation to the posterior distribution for the first Gaussian process distribution with variational parameters, and providing an approximation for the first Gaussian process distribution as the first Gaussian process distribution, and / or, the second Gaussian process distribution includes a posterior distribution, and the method includes the steps of learning a particularly sparse approximation to the posterior distribution for the second Gaussian process distribution with variational parameters, and providing an approximation for the second Gaussian process distribution as the second Gaussian process distribution. In this way, the parameters defining the Gaussian process distribution are learned for an unknown function.

[0010] The method preferably includes the step of determining a first or second Gaussian process distribution having an expected likelihood term that is output-dependent and decomposes between agents and between time points. The likelihood term is an approximation of a portion of the lower bound of evidence (ELBO) that allows for the determination of parameters defining the Gaussian process distribution.

[0011] The method may also be applied to a quadratic ordinary differential equation, wherein the latent state of the first agent includes a first component and a second component, and the method includes the step of changing the latent state of the first component of the first agent in response to the change in the second component of the latent state of the first agent, wherein the second component changes in response to the first contribution and the second contribution to the change in the latent state of the first agent.

[0012] To control the first agent, the method may include a step of determining the action of the first agent in response to the output.

[0013] The method preferably includes the step of determining the latent state of a first agent and the latent state of a second agent based on measurements of a series of observable states of the agent.

[0014] The first agent may be an existing object in the physical world, and the latent state of the first agent is determined according to measurements of the first agent's properties, and / or the second agent may be an existing object in the physical world, and the latent state of the second agent is determined according to measurements of the second agent's properties, in particular the measurements include positional data, in particular positional data from satellite navigation systems, or in particular digital images, preferably video images, radar images, LiDAR images, ultrasonic images, motion images, and / or thermal images, preferably positional data from information regarding the agent's position or velocity. This makes it possible to determine the latent state from measurements, in particular appropriate control of the first agent.

[0015] A device for continuous-time interaction modeling of agents comprises at least one processor and at least one memory configured to perform the steps of this method. This device has advantages corresponding to the advantages of this method.

[0016] The device preferably includes an interface configured to observe the agent's continuous-time interaction, particularly digital images, preferably video images, radar images, LiDAR images, ultrasound images, motion images, and / or thermal images, or to receive information regarding the agent's continuous-time interaction.

[0017] The interface may be configured to control at least one action of the agents in response to an output or action.

[0018] A computer program that, when executed by a computer, contains computer-readable instructions that cause the computer to perform the method, provides advantages corresponding to the advantages of this method.

[0019] Further advantageous embodiments are derived from the following description and drawings. [Brief explanation of the drawing]

[0020] [Figure 1] This diagram schematically shows a device for modeling the continuous-time interaction of agents. [Figure 2] This figure shows the steps in a method for modeling continuous-time interactions between agents. [Modes for carrying out the invention]

[0021] Figure 1 shows device 100 for continuous-time interaction modeling of agent 102. Figure 1 shows four agents 102. There may be more or fewer agents 102 than four.

[0022] Device 100 comprises at least one processor 104 and at least one memory 106. Device 100 may also include an interface 108.

[0023] At least one processor 104 is configured to perform steps of the method described later. At least one memory 106 is configured to store instructions, in particular computer programs, that, when executed by at least one processor 104, cause the processor 104 to perform steps of the method.

[0024] Interface 108 is configured, in one example, to observe the continuous-time interactions of agent 102 or to receive information regarding the continuous-time interactions of agent 102. Interface 108 is also configured, in one example, to control at least one action of agent 102.

[0025] The information is provided, for example, in the form of digital images, video images, radar images, LiDAR images, ultrasound images, motion images, and / or thermal images.

[0026] The continuous-time interaction of agent 102 includes, for example, the position or velocity of agent 102. The position may be a relative position, for example, the distance between a pair of agents 102, or the absolute position of agent 102.

[0027] In this example, system 110 includes agent 102. System 110 may be a physical system. Agent 102 may be a physical system, in particular a technical system. Agent 102 may be an existing real object in the physical world. Agent 102 may include a vehicle, a pedestrian, or other moving object such as a ball.

[0028] System 110 includes an environment 112. Environment 112 may include road infrastructure or building infrastructure. Agent 102 in this example moves within environment 112 and may be affected by environment 112. Agent 102 may include objects of environment 112, such as fixed infrastructure systems that are part of environment 112. System 110 in this example follows certain physical rules. One such physical rule is that the integral of velocity is position.

[0029] Continuous-time interactions are not limited to these physical quantities. They may include other physical, technical, or chemical quantities. Continuous-time interactions may also include global latent variables, such as agent-specific properties like mass or radius.

[0030] Figure 2 shows the steps in a method for modeling the continuous-time interaction of agents.

[0031] This method uses the model:

number

number

number

number

number

number

number

number

number

number

number

number

[0032] The model consists of two independent additive functions, i.e., kinematic functions.

number

number

[0033] The kinematic function f in this example s This learns how the agent behaves over time when no other agents are present and therefore independent of other agents. The interaction function f in this example b The agents learn how to interact with each other.

[0034] This method is described for multiple agents a (a=1,…,A).

[0035] In one example, the method involves, for each agent a, A terms, i.e., the kinematic function f sThe step involves determining the dynamics according to the sum of one independent kinematic term that depends on and, in this example, A-1 interaction terms, each of which is an interaction function f b This method models the interaction between agent a and one of the remaining A-1 agents a'. This method is not limited to determining the dynamics of each agent a. This method may also include a step to determine the dynamics of a subset of multiple agents (a=1,...,A) or a single agent a from multiple agents (a=1,...,A). This method is not limited to determining the dynamics of agent a in relation to its interaction with the remaining A-1 agents a'. This method is applicable to a subset N of multiple agents (a=1,...,A). a The process may include a step of determining the dynamics of agent a, depending on the situation, or depending on one agent a' among multiple agents (a=1,…,A). Subset N a In one example, this is the neighborhood of agent a.

[0036] This method is for a dataset of P sequences.

number

number

number

[0037] This method includes step 200.

[0038] In step 200, the data sequence

number

[0039] In the exemplary method, agent 102 of system 110 is observed over a period of time. In this example, multiple agents a (a=1,…,A) represent agent 102, and the data sequence

number

number

[0040] Data Sequence

number

[0041] The observable state may be determined in particular from digital images of the system 110, preferably video images, radar images, LiDAR images, ultrasound images, motion images, and / or thermal images. These include, in this example, information regarding the position and velocity of agent 102.

[0042] This method includes step 202.

[0043] In step 202, the initial latent state of at least one agent a at start time t1 is determined.

number

[0044] At least one initial latent state of agent a

number

number

number

number

number

number

number

[0045] In the exemplary method, the initial values ​​of agent a required for the integration of the ordinary differential equation are determined. These initial values ​​can generally represent a variety of things. In this example, the initial values ​​correspond to the initial position and velocity of each agent a. In this example, the initial values ​​represent the initial latent state of system 110 at start time t1.

number

[0046] In this example, encoder q Θis a combination of a recurrent neural network and a multi-perceptron. Another exemplary encoder includes a graph neural network layer to capture interactions. Encoder q Θ can also be another neural network architecture. Encoder q Θ is configured to output a distribution rather than a single value.

[0047] If the model includes global latent variables, such as object-specific characteristics like mass or radius, another encoder may be used to extract these variables. Both encoders may have the same architecture. Both encoders output a distribution rather than a single value.

[0048] This method includes step 204.

[0049] In step 204, a first function f for modeling the kinematic behavior of agent a independently of other agent a' s is provided with a first Gaussian process distribution

Number

[0050] In step 204, a second function f for modeling the interaction between agents a, a' b is provided with a second Gaussian process distribution

Number

[0051] The kinematic function f s and the interaction function f b are unknown in one example. The first Gaussian process distribution

Number

number

number

number

number

number

number

number

[0052] This method includes step 206.

[0053] In step 206, the first function f s However, the first Gaussian process distribution

number

[0054] In step 206, the second function fb However, the second Gaussian process distribution

number

[0055] First function f s This is the latent state h of one agent a. a (τ) and its latent state h a Contribution to the change in (τ) f s (h a (τ)) It is configured to map to [something].

[0056] Second function f b This is the latent state of at least two agents a and a'.

number

number

[0057] kinematic function f s and interaction function f b The kinematic function f is defined in continuous time using ordinary differential equations. s and interaction function f b This corresponds to the time derivative.

[0058] This method includes step 208.

[0059] In step 208, at time t n The latent state h of at least one agent a in a (t n ) will be decided.

[0060] time tn The latent state h of at least one agent a in a (t n The step of determining the initial latent state

number

number

[0061] The resulting latent state h of agent a from this change a (t n ) is determined, for example, as follows:

number

[0062] In one example, the latent state h is considered for multiple agents (a=1,…,A). a (t n ) will be decided.

[0063] In one example, this method involves a subset N of multiple agents (a=1,…,A) including agent a. a The step of selecting and subset N a This includes the step of determining how agent a changes in response to other agents a' within the subset N. Thus, only agents in the vicinity of agent a are used. a Agent a' within the system is selected, for example, based on a measure of distance between agents.

[0064] In one example, this method is used for subset N a Located within, subset N aDepending on at least one agent a' that is independent of at least one of the multiple agents (a=1, ..., A) outside of the set, at time t n The latent state h of at least one agent a in a (t n This includes a step of changing the latent state h of one agent a that results from this change. a (t n ) is determined, for example, as follows:

number

[0065] This method includes step 210.

[0066] Step 210 includes the step of determining the output of the model.

number

[0067] The output in this example is the trajectory sample for each agent a.

number

[0068] For inference purposes, this method may terminate, or produce another output, for example, another data sequence.

number

[0069] The output may be used to determine an action for at least one agent 102. The path taken by agent 102 is determined, for example, in accordance with the positions of other agents to avoid collisions. For example, the action is to send an instruction to agent 102 to move to a target position. The instruction may be sent to agent 102 but executed by agent 102 as its own action. In one example, agent 102's action is determined according to the output.

[0070] For training purposes, this method may include step 212.

[0071] Step 212 is the first Gaussian process distribution

number

number

[0072] In one example, during training, the first Gaussian process distribution

number

number

number

[0073] In one example, during training, the second Gaussian process distribution

number

number

number

[0074] Training is the lower limit of evidence (ELBO). logp(Y 1:N )≧E q [logp(Y 1:N │H1,f,U)]-KL[q(H1)||p(H1)]-KL[q(U)||p(U)] It may also be used as an optimization target to maximize, and in the formula, H1~q Φ (H1|Y 1:N ) and Φ are parameters of a neural network encoder that outputs a Gaussian distribution with diagonal covariance, and p(H1)=N(0,I) is a prior standard Gaussian distribution with dimensions suitable for the initial latent state.

number

number

number

number

number

number

[0075] Training may also include learning the parameters Φ of the neural network encoder, and other parameters such as kernel variance or noise variance.

[0076] This ELBO does not have a closed-form representation, therefore, the orbital sample

number

number

[0077] The term KL[·] corresponds to the Kullback-Leibler regularizer. The prior distribution on the induced variable is the Gaussian process p(U)=p(U s )p(U b ) Accordingly, here,

number

number

[0078] In one example, the model includes a second-order ordinary differential equation. h a (t)≡[s a (t),v a (t)]

number

[0079] This is the latent state h of agent a. a (t) is the first component s a (t) and the second component v a This means that (t) is included. This method is the second component v of the latent state of agent a. a (t), and the second component v of agent a's latent state a Changes with respect to (t)

number

number

[0080] This eliminates other unidentifiable issues and latent states s a (t), v a This improves the interpretation of (t) and thus eases the reasoning. In this example, s a (t) is a latent state depending on the position, v a (t) is the latent state corresponding to the velocity of agent a. Therefore, an embodiment of this method is as follows.

[0081] Second-order ordinary differential equations perform significantly better than first-order ordinary differential equations when there is missing data.

Claims

1. A computer-implemented method for continuous-time interaction modeling of agents, The aforementioned method, Step (202) provides the latent state of a first agent and the latent state of a second agent, and in particular characterizes the position or velocity of the first and second agents (102), Step (204) provides a first Gaussian process distribution with respect to a first function for modeling the kinematic behavior of an agent independently of other agents, and a second Gaussian process distribution with respect to a second function for modeling the interaction between agents. Step (206) of sampling the first function from the first Gaussian process distribution and the second function from the second Gaussian process distribution, Includes, The first function is configured to map the latent state of one agent to a first contribution to the change in the latent state of the one agent, The second function is configured to map the latent states of the two agents to a second contribution to the change in the latent state of one of the two agents. The aforementioned method, (208) A step of changing the latent state of the first agent in accordance with the first contribution resulting from the mapping of the latent state of the first agent to the first contribution by the first function and the second contribution resulting from the mapping of the latent state of the first agent and the latent state of the second agent to the second contribution by the second function. Methods that include...

2. The aforementioned method, Step (202) provides the initial latent state of a plurality of agents, including the first agent and the second agent, The steps include: changing the latent state of the first agent according to the second contribution resulting from mapping the pairs of latent states of the first agent and a different second agent among the plurality of agents using the second function (208); or selecting a subset of the plurality of agents (208) and changing the latent state of the first agent according to the second contribution resulting from mapping the pairs of latent states of the first agent and a different second agent of the subset using the second function (208); The method according to claim 1, including the method described in claim 1.

3. The aforementioned method, A step (208) of determining an agent from the plurality of agents with respect to the subset of the plurality of agents that is closer to the first agent than the other agents, according to a measure of distance between agents. The method according to claim 2, including the method described in claim 2.

4. The aforementioned method, Data sequence ( [Math 1] (200) step of providing Includes, The step (202) of providing the initial latent state of the first agent and / or the second agent includes determining the initial latent state of the data sequence using an encoder configured to map the data sequence to the initial latent state of the data sequence. The method according to claim 2.

5. The aforementioned method, Step (210) of determining the output according to the latent state of the first agent, in particular a trajectory sample, preferably the trajectory of the position and / or velocity of the first agent over time. The method according to claim 1, including the method described in claim 1.

6. The first Gaussian process distribution described above includes the posterior distribution, The aforementioned method, The first Gaussian process distribution with variational parameters is to learn a particularly sparse approximation to the posterior distribution (212), The first Gaussian process distribution is provided as the approximation (204), Includes, and / or, The second Gaussian process distribution described above includes the posterior distribution, The method includes the step (212) of learning a particularly sparse approximation of the posterior distribution for the second Gaussian process distribution with variational parameters, The step (204) of providing the approximation for the second Gaussian process distribution as the second Gaussian process distribution, including, The method according to claim 5.

7. The aforementioned method, (212) A step of determining the first Gaussian process distribution or the second Gaussian process distribution having expected likelihood terms that depend on the output and are decomposed between agents and between time points. The method according to claim 5 or 6, including the method described in claim 5 or 6.

8. The latent state of the first agent comprises a first component and a second component, The aforementioned method, Steps to change the latent state of the first component of the latent state of the first agent in response to the second component of the latent state of the first agent and the change in the second component of the latent state of the first agent. Includes, The second component of the latent state of the first agent changes according to the first contribution to the change of the latent state of the first agent and the second contribution to the change of the latent state of the first agent. The method according to claim 1.

9. The aforementioned method, The step of determining the action of the first agent in accordance with the output. The method according to claim 5, including the method described in claim 5.

10. The aforementioned method, Steps (200, 202) to determine the latent state of the first agent and the latent state of the second agent in accordance with a series of observable state measurements of the agent (102). The method according to claim 1, including the method described in claim 1.

11. The first agent is an existing object in the physical world, The latent state of the first agent is determined according to the measured values ​​of the first agent's characteristics (200), and / or, The second agent is an existing object in the physical world, The latent state of the second agent is determined according to the measured properties of the second agent (202) In particular, the measured values ​​include position data, especially position data from a satellite navigation system, or especially digital images, preferably video images, radar images, LiDAR images, ultrasonic images, motion images, and / or thermal images, preferably position data from information regarding the position or velocity of the agent (102). The method according to claim 10.

12. A device (100) for continuous-time interaction modeling of agents, A device (100) comprising at least one processor (104) and at least one memory (106) configured to carry out the steps of the method according to claim 1.

13. The device (100) according to claim 12, comprising an interface (108) configured to capture a series of observable state measurements, particularly digital images, preferably video images, radar images, LiDAR images, ultrasonic images, motion images, and / or thermal images, to observe the continuous-time interaction of the agent (102), or to receive information relating to the continuous-time interaction of the agent (102).

14. The device (100) according to claim 13, wherein the interface (108) is configured to control at least one action of the agent (102) in response to an output determined by the method of claim 5 or an action determined by the method of claim 9.

15. A computer program that includes computer-readable instructions for causing a computer to perform the method described in claim 1 when executed by the computer.

Images