Computer-implemented method for predicting behavior of agent in kinetic system comprising many agents interacting with each other

Abstract

To provide a computer-implemented method for predicting a behavior of an agent.SOLUTION: A value of first moment of a first distribution for modeling an agent latent state is identified for every component, a value of second moment of the first distribution is identified, and for every component, an expectation value to the first moment of a second distribution to a prediction time point is identified in relation to the value of the first moment of the first distribution and a value of the second moment to the prediction time point. The second distribution models a behavior of the agent in relation to the agent latent state, the expectation value to the first moment of the second distribution regulates first moment of a third distribution. For every component, the second moment of the third distribution is identified. Especially, a final total of the third distribution of the component which is weighted by at least one weight, is identified, and related to the final total, prediction of the behavior is identified.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method for predicting the behavior of agents in a dynamic system comprising a number of interacting agents.

[0002] Charlie Tang, Russ Salakhutdinov “Multiple Futures Prediction” 2019, NeurIPS and Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, Raquel Urtasun “Implicit Latent Variable Model for Scene-Consistent Motion Forecasting” 2020, ECCV disclose methods for predicting the behavior of such systems.

[0003] Disclosure of the invention The computer-implemented method and apparatus described in the independent claim can accurately predict the behavior of an agent while keeping the cost of the computational resources required for the prediction low.

[0004] In a dynamic system comprising numerous interacting agents, a method for predicting agent behavior in relation to the agent's latent state assumes that, for multiple components and multiple time points up to the prediction time, for each component, the value of the first moment of a first distribution that models the agent's latent state is identified, the value of the second moment of the first distribution is identified, for each component, the expected value for the first moment of the second distribution up to the prediction time is identified in relation to the value of the first moment of the first distribution up to the prediction time, and in relation to the value of the second moment of the first distribution up to the prediction time, the second distribution models the agent's behavior in relation to the agent's latent state, the expected value for the first moment of the second distribution defines the first moment of a third distribution, for each component, the second moment of the third distribution is identified, and in particular, the sum of the third distribution of the components, weighted by at least one weight, is identified, and the prediction of behavior is identified in relation to this sum.

[0005] Preferably, the value of the first moment of the first distribution is identified in relation to the value of the first moment of the first distribution for a preceding time point and the expected value of the deterministic change of the first moment of the first distribution, and / or the value of the second moment of the first distribution for this time point is identified in relation to the value of the second moment of the first distribution for a preceding time point, the covariance of the deterministic change and the expected value of the stochastic change of the second moment of the first distribution. This allows each value to be identified efficiently and recursively.

[0006] The value of the second moment of the first distribution for this point in time is preferably determined in relation to the value of the second moment of the first distribution for a preceding point in time, the covariance of the deterministic change, the covariance of the latent state for the preceding point in time with the deterministic change, the transpose of the covariance of the latent state for the preceding point in time with the deterministic change, and the expected value for the stochastic change. This allows the value to be determined efficiently and recursively.

[0007] Preferably, the expected value for the first moment of the second distribution is determined in relation to the value of the first moment of the first distribution relative to the prediction time. This allows for efficient determination of the expected value.

[0008] Preferably, for each component, the covariance of the first moment of the second distribution is identified in relation to the value of the first moment of the second distribution with respect to the predicted time; for each component, the expected value of the second moment of the second distribution with respect to the predicted time is identified in relation to the latent state with respect to the predicted time; and for each component, the second moment of the third distribution is identified in relation to the covariance of the first moment of the second distribution and the expected value of the second moment of the second distribution with respect to the predicted time. This makes the method particularly efficient.

[0009] Preferably, a context variable is identified, the context variable includes an assignment, the assignment assigns at least one agent to another agent to be considered for predicting the behavior of this agent, and / or the context variable characterizes the history of a dynamic system. The first moment of a first distribution is identified in relation to the context variable, and / or the second moment of the first distribution is identified in relation to the context variable, and / or the expected value for the first moment is identified in relation to the context variable, and / or the first moment of a third distribution is identified in relation to the context variable, and / or the second moment of the third distribution is identified in relation to the context variable, and / or at least one weight is identified for at least one component in relation to the context variable. This takes into account the agent's neighborhood and history.

[0010] The history is preferably identified in relation to the observed behavior of at least one agent, particularly in relation to behavior including the position or movement of the agent, the position or movement of the agent being detected in particular by a receiver for a satellite-supported positioning system, or at least one digital image being detected in particular by a sensor for digital imaging, preferably a camera, LiDAR sensor, ultrasonic sensor, motion sensor, infrared imaging sensor and / or radar sensor, the position or movement of the agent being identified in relation to at least one digital image, or a signal being detected by a speaker receiving audible sound, the position or movement of the agent being identified in relation to the signal.

[0011] It is often assumed that the context variable contains a matrix, where each row of the matrix represents one of these agents, and each column of the matrix represents one of these agents, and at least one of the elements identified by the rows and columns of the matrix, in particular a binary value, is identified, which sets whether the agent identified by the row should be considered for predictions for the agent identified by the column. Or, at least one of the elements identified by the rows and columns of the matrix, in particular a binary value, is identified, which sets whether the agent identified by the column should be considered for predictions for the agent identified by the row. The matrix represents the neighborhood to be considered. This ensures that the most relevant agents are considered in the computation, in particular. This ensures that the best possible predictions are computed particularly efficiently.

[0012] Preferably, the first and second moments of the first distribution are identified iteratively, and for the first iteration of these iterations, the values ​​of the first moment and the second moment of the first distribution, related to the context variable, are identified for each component. This allows for particularly efficient consideration of history.

[0013] Preferably, for prediction purposes, it is assumed that multiple latent states of one agent are modeled independently of each other, and multiple latent states of various agents are modeled independently of each other; or multiple latent states of one agent are modeled independently of each other, and corresponding elements of multiple latent states of various agents are modeled in relation to each other; or various elements of one latent state of one agent are modeled in relation to each other, and multiple latent states of various agents are modeled independently of each other. This makes the computation extremely efficient.

[0014] Preferably, in relation to the prediction, at least one agent, in particular a computer-controlled machine, in particular a robot, preferably a vehicle, household appliance, driven machine, manufacturing machine, personal assistant, or access control system is driven and controlled. This drive control is particularly robust.

[0015] At least one agent may be a real object that exists in the physical world.

[0016] The apparatus includes at least one processor and at least one memory, the at least one processor and at least one memory being configured to carry out this method. The apparatus has advantages corresponding to the advantages of the method.

[0017] The system includes at least one agent, in particular a computer-controlled machine, in particular a robot, preferably a vehicle, household appliance, driven machine, manufacturing machine, personal assistant, or access control system, wherein the agent or system includes a device, the device configured to drive and control the agent in relation to prediction. The system has advantages corresponding to the advantages of the method.

[0018] A computer program contains computer-readable instructions, and this method is implemented when the computer executes the instructions. This computer program has advantages commensurate with the advantages of the method.

[0019] Other advantageous embodiments will become apparent from the following specification and drawings. [Brief explanation of the drawing]

[0020] [Figure 1] This diagram schematically illustrates a device that predicts the behavior of agents in a dynamic system comprising numerous interacting agents. [Figure 2] This diagram illustrates the behavior of an agent in an exemplary dynamic system. [Figure 3] This figure shows predictions about the behavior of agents in a dynamic system. [Figure 4] This diagram shows the steps involved in the prediction method. [Figure 5] Figures 5a to 5d show examples of neural networks. [Figure 6] This figure schematically illustrates the approximation of the covariance matrix.

[0021] Figure 1 schematically shows a device 100 that predicts the behavior of agents 102 in a dynamic system 104 comprising a number of interacting agents 102. In this example, the dynamic system 104 is a physical system, in particular a technical system. Agents 102 may be physical systems, in particular technical systems. Agents 102 may be actual objects existing in the physical world. The device 100 includes at least one processor 106 and at least one memory 108. The device 100 is configured to implement a method for predicting the behavior of agents 102 in the dynamic system 104, as described below. The device 100 optionally includes an interface 110. Agents 102 optionally include an interface 112. The device 100 and agents 102 are optionally configured to communicate through their interfaces, for example, by which information about the behavior of agent 102 is transmitted from agent 102 to the device 100, or information about the prediction of behavior is sent from the device 100 to agent 102. A sensor device 114 may be provided, and the sensor device 114 is configured to detect information regarding the behavior of agent 102 in a dynamic system 104. The sensor device 114 may be configured to measure the physical characteristics of agent 102. Agent 102 is optionally configured to provide information regarding its own behavior and information regarding the behavior of other agents 102. For example, information regarding its own behavior is detected by the sensor device 114. For example, the sensor device 114 is placed on one or more agents 102 and is configured to detect information regarding the behavior of each agent 102 and / or the behavior of other agents 102. The sensor device 114 is configured, for example, to detect the position or movement of agent 102.The sensor device 114 may include a receiver for a satellite-supported positioning system, such as a global positioning system, or a sensor for digital imaging, such as a camera, LiDAR sensor, ultrasonic sensor, motion sensor, infrared imaging sensor, and / or radar sensor. The sensor device 114 is configured, for example, to detect the position or movement of agent 102. The sensor device 114 may include a speaker that receives audible sound and generates an audio signal. It may be assumed that the sensor device 114 is located in infrastructure 116 to which agent 102 can move and is connected to the interface 110 of device 100 at least temporarily via a communication connection 118. Instead of the sensor device 114, data containing information about agent 102 may be provided, in particular graph-structured data.

[0022] Agent 102 is optionally configured to identify its own behavior in relation to predictions about the behavior of other agents 102. Each agent 102, for example, includes one actuator 120, which is configured to influence the behavior of each agent 102 in relation to the prediction. It is also possible that the device 100 is configured to identify a drive control command for at least one agent 102 in relation to the prediction, rather than transmitting the prediction to the agents 102, and to transmit the drive control command to one or more agents 102 to be driven. In this case, the actuator 120 is configured to execute the drive control command. It is also possible that the device 100 is integrated with one or more agents 102.

[0023] Similarly, a computer program is provided that includes instructions for the computer to perform a method when the computer runs it. For example, at least one processor 102 executes the computer program.

[0024] Figure 2 shows the behavior of agent 102 in an exemplary system 104. In this example, the behavior of agent 102 is observed, and Figure 2 shows the actual trajectory that agent 102 moved from the start of the observation to the end of the observation, in accordance with the observation of its own behavior.

[0025] The dynamic system 104 is, for example, a technical system where agent 102 is a computer-controlled machine, such as a robot such as a vehicle, household appliance, driven machine, manufacturing machine, personal assistant, or access control system. The dynamic system 104 may be any other system. For example, the dynamic system 104 is molecular dynamics where agent 102 is an atom or molecule and its movement is predictable. For example, the dynamic system is a sport, such as a soccer game, a basketball game, or an American football game, where the agent is a human or a sporting object, such as a ball, and its movement is predictable.

[0026] The dynamic system 104 in this example is a roundabout 202. In this example, the roundabout 202 has a first entrance 204, a second entrance 206, a third entrance 208, and a fourth entrance 210. In this example, the roundabout 202 has a first exit 212, a second exit 214, a third exit 216, and a fourth exit 218. Agent 102 in this example includes a vehicle. It may be assumed that Agent 102 also includes pedestrians. The first vehicle moves from the first entrance 204 through the roundabout 202 on the first observed trajectory 220 from the start, and at the end, is located within the roundabout 202 in the area of ​​the second exit 214. The second vehicle, starting from the beginning, moves along the second observed track 222, from the area of ​​the second exit 214, through the roundabout 202, exits the roundabout 202 via the third exit 216, and is outside the roundabout 202 at the end. The third vehicle, starting from the beginning, moves along the third observed track 224, from the second entrance 206, through the roundabout 202, and is inside the roundabout 202, in the area of ​​the third exit 216, at the end. The fourth vehicle, starting from the beginning, moves along the fourth observed track 226, from the area of ​​the roundabout 202 between the second exit 214 and the second entrance 206, through the roundabout 202, and is at the fourth exit 218 at the end. The fifth vehicle moves from the starting point along the fifth observed track 228, within the area of ​​the roundabout 202 between the third entrance 208 and the fourth exit 218, and is located at the first exit 212 at the end point. The sixth vehicle moves from the starting point to the end point along the sixth observed track 230, within the area of ​​the fourth entrance 210.

[0027] Figure 3 shows a prediction of the behavior of agent 102 in a dynamic system 104, based on the example of a roundabout 202.

[0028] The first vehicle moves along the first observed trajectory 220 from the start to the end of the observation. In this example, the first vehicle has not moved by the end of the observation and is located at the first entrance 204. Between the end of the observation and the end of the prediction, a first predicted trajectory 320 is identified for the first vehicle. According to this prediction, the first vehicle moves from the first entrance 204 through the roundabout 202 and, at the end, is located within the roundabout 202 in the area of ​​the second exit 214.

[0029] The second vehicle moves along the second observed track 222 from the start to the end of the observation, up to the area of ​​the roundabout 202 between the second entrance 206 and the third exit 216. This section of the second observed track 222 is shown as a dashed line in Figures 2 and 3. Between the end of the observation and the end of the prediction, a second predicted track 322 is identified for the second vehicle. According to this prediction, the second vehicle moves from the area of ​​the roundabout 202 between the second entrance 206 and the third exit 216, through the roundabout 202, exits the roundabout 202 via the third exit 216, and is outside the roundabout 202 at the end of the observation.

[0030] The third vehicle moves along the third observed trajectory 224 from the start to the end of the observation. In this example, the third vehicle has not moved by the end of the observation and is located at the second entrance 206. Between the end of the observation and the end of the prediction, a third predicted trajectory 324 is identified for the third vehicle. According to this prediction, the third vehicle moves from the second entrance 206 through the roundabout 202 and, at the end, is located within the roundabout 202 in the area of ​​the third exit 216.

[0031] The fourth vehicle moves along the fourth observed track 226 from the start to the end of the observation, up to the area of ​​the roundabout 202 between the third exit 216 and the third entrance 208. This section of the fourth observed track 226 is shown as a dashed line in Figures 2 and 3. Between the end of the observation and the end of the prediction, the fourth predicted track 326 is identified for the fourth vehicle. According to this prediction, the fourth vehicle moves from the area of ​​the roundabout 202 between the third exit 216 and the third entrance 208, through the roundabout 202, and is located at the fourth exit 218 at the end of the observation.

[0032] The fifth vehicle moves along the fifth observed track 228 from the start to the end of the observation, to the area of ​​the roundabout 202 between the fourth exit 218 and the fourth entrance 210. This section of the fifth observed track 228 is shown as a dashed line in Figures 2 and 3. Between the end of the observation and the end of the prediction, the fifth predicted track 328 is identified for the fifth vehicle. According to this prediction, the fifth vehicle moves from the area of ​​the roundabout 202 between the fourth exit 218 and the fourth entrance 210, through the roundabout 202, and is located at the first exit 212 at the end of the observation.

[0033] The sixth vehicle moves along the sixth observed trajectory 230 from the start to the end of the observation. In this example, the sixth vehicle has not moved by the end of the observation and is still at the fourth entrance 210. Between the end of the observation and the end of the prediction, the sixth predicted trajectory 330 is identified for the sixth vehicle. The sixth vehicle moves within the area of ​​the fourth entrance 210 until the end, according to this prediction.

[0034] The prediction, or in this example, the predicted trajectory, is approximated as a Gaussian mixture distribution. The moments of the Gaussian mixture distribution are identified for each individual agent 102 in relation to the observed portion of its own behavior, i.e., in this example, the observed portion of each observed trajectory, shown by the dashed line, by the method described below.

[0035] In this example, a 95% confidence interval is visualized for predictions regarding another illustrated portion of the observed trajectory.

[0036] Predicting trajectories, i.e., predicting the temporal progression of the position of agent 102, is one example. It may also be assumed that the distance between these agents 102, and the velocity or acceleration of agent 102, are specified.

[0037] The behavior of agent 102, in this example the behavior of the vehicles, is observed for a predetermined period of time. Predictions are identified in relation to the behavior observed for the predetermined period of time. In one example, at least one of the vehicles is an autonomous vehicle. The prediction is a simulation of the surroundings of this at least one autonomous vehicle, and the at least one autonomous vehicle is driven and controlled in relation to this prediction.

[0038] In this example, prediction identification is performed by machine learning of a model, and the prediction is identified by this model.

[0039] This will be discussed later.

number

number

number

number

number

Number

Number

Number

Number

Number

Number

number

number

[0040] variable

number

number

[0041] The predictions are then specified for M agents 102, denoted by m.

[0042] In contrast, deterministic change is

number

number

number

number

number

number

number

number

number

[0043] After t prediction steps, the model considers the correlation between agents 102 that are separated from each other by a distance of up to t steps. In one example, this distance represents how many edges must be traversed to reach agent m' from agent m. This distance can be infinite if the agents are not connected to other agents.

[0044] The prediction for prediction time T is given by the marginal probability p(y T |I) is such that this is a probability p(y T |x T ), kernel p(x T Nested integrals with |x0,I) and a Gaussian mixture model, GMM, p(x0|I).

number

[0045] kernel p(x T |x0,I) is the mean μ for each time step t. t (I) and covariance Σ t (I) A normal distribution N(x t |μ t (I),Σ t (I)) is approximated by, where

number

[0046] In this example, the function f(x,I) is implemented as a neural network. The function L(x,I) is implemented as a neural network. The function g(x) is implemented as a neural network. The function Q(x) is implemented as a neural network.

[0047] The expected values ​​and covariances for the output of each neural network are identified, for example, as described in Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E. Turner, Jose Miguel Hernandez-Lobato, Alexander L. Gaunt: “Deterministic Variational Inference for Robust Bayesian Neural Networks”, in ICLR, 2019a (Anqi Wu).

[0048] Mutual covariance Cov[x t ,f(x t ,I) is, for example,

number

number

number

[0049] Figure 4 shows the behavior of agent m in a dynamic system 104 comprising a large number of M interacting agents m.

number

[0050] Prediction p(y T |I) is the latent state x of agent m. t This is identified in relation to the method, which involves two loops: an inner loop and an outer loop.

[0051] During the first of these iterations, the latent state x0 is derived from a Gaussian mixture model with V components v, defined by a normal distribution N(x0|μ0(I),Σ0(I)).

[0052] Normal distribution N(x t |μ t (I),Σ t (I)) the first moment μ t and the second moment Σ t This is determined in the inner loop, in the iterations. In the initial state, each component v is distributed according to a normal distribution N(x). 0,v |μ 0,v (I),Σ 0,v (I)) the first moment μ 0,v The value of and the second moment Σ 0,v These are the values. In this example, these values ​​are related to the context variable I.

[0053] Moment μ 0,v and Σ 0,v The value of is determined by another neural network in relation to the context variable I. Figure 5a shows an example of this neural network, comprising 30 fully connected layers and Tanh activations, followed by a layer for operation AGG, followed by 64 fully connected layers and Tanh activations, followed by the first moment μ 0,v A fully connected layer follows the value of , followed by another fully connected layer with Exp activation.

[0054] The inner loop is calculated at the predicted time T for multiple V components v and multiple time points t. The outer loop is calculated at the predicted time T for multiple V components v.

[0055] Normal distribution N(x t |μ t (I),Σ t (I)) is the latent state x of agent m. t We model the normal distribution N(y t |g(x t ), QQ T (x t )) is the behavior of agent m y t The latent state x of agent m t In relation to this, we will model it.

[0056] In this method, the normal distribution N(a T,v (I), B T,v (I)) models the behavior of individual components v.

[0057] In step 402, context variable I is set. In one example, context variable I is assigned N m This includes this assignment N m This assigns at least one agent m to another agent m that should be considered for predicting the behavior of agent m. The context variable I is given in this example.

[0058] Assignment N m In some examples, this is a matrix, where each row of the matrix represents one of the agents m, and each column of the matrix represents one of the agents m.

[0059] In this example, a specific binary value is identified for each element of the matrix.

[0060] In one example, the values ​​of the elements identified by their own rows and columns in a matrix determine whether the agent m identified by the row should be considered for predictions regarding the agent m identified by the column.

[0061] In one example, the values ​​of elements identified by their own rows and columns in a matrix determine whether the agent m identified by the column should be considered for predictions regarding the agent m identified by the row.

[0062] For example, the relationships between agents m are modeled by a graph, where the edge values ​​ε are determined such that agents m' adjacent to agent m in the graph are taken into consideration in their own predictions.

[0063] In this example, context variable I contains the history of the dynamic system 104.

[0064] In this example, the context variable I is used to identify the moment, expected value, and weight, and its argument contains the context variable I.

[0065] In this example, several V components v are identified by a neural network, and the input to the neural network includes the dynamic history of system 104 and edges ε from context variable I. In one example, the history of system 104 is defined by the observed behavior of agent m, particularly by the observed portion of its trajectory.

[0066] In this example, the edge is matrix N m The binary value in is 0 or 1, where a value of 1 indicates that an edge exists between the two nodes, and a value of 0 indicates that no edge exists between the two nodes. The latent state of agent m at time t.

number

[0067] In some cases, the track is defined by a temporal sequence of two- or three-dimensional geographical coordinates, which presents a temporal sequence of the vehicle's position.

[0068] Operation AGG, in this example, messages

number

number

[0069] A neural network is, for example, a graph neural network. This is constructed as described, for example, in Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Guelcehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, MatthewBotvinick, Oriol Vinyals, Yujia Li, Razvan Pascanu. “Relational inductive biases, deep learning, and graph networks” arXiv, abs / 1806.01261, 2018.

[0070] In step 404, for multiple V components v and multiple time points t=1,...T, iteratively, for each component v, a normal distribution N(x) is calculated up to the predicted time point T. t |μ t (I),Σ t (I)) the first moment μ t The value is determined.

[0071] First moment μ t The value of is determined recursively in this example. This is the first moment μ at time t. t The value of is the first moment μ with respect to a time point preceding time t, for example, t-1. t-1 This means it is identified in relation to the value of .

[0072] The following explanation is based on tools that can determine the expected value E[f(x)], the covariance matrix Cov(f(x)), and the cross-covariance matrix Cov(x,f(x)) of a function f(x). The expected value E[f(x)] and the covariance matrix Cov(f(x)) are determined, for example, as described by Anqi Wu. The cross-covariance matrix Cov(x,f(x)) is determined, for example, as described by Andreas Look.

[0073] The tool requires that in a neural network, layers are used whose moments can be computed on the output side in order to determine the expected value E[f(x)], the covariance matrix Cov(f(x)), and the cross-covariance matrix Cov(x,f(x)). For this purpose, operation AGG(x t ,ε) is used.

[0074] Operation AGG(x t For example, ε) is performed in each neural network as an average aggregation, where the message is applied to layer l of the neural network.

number

number

[0075] For example, the Kronecker product

number

number

number

[0076] This tool requires that the same affine transformation be performed on these layers l of the neural network by the same weight matrix W l and the same bias b l . This calculation is, in one example, common to all layers and is done by the Kronecker product, [Math.] where [Math.] and where the Jacobian matrix can be used as [Math.] .

[0077] The value of the first moment μ t is, in this example, specified for the deterministic change f(x t-1 , I) of the first moment μ t in connection with the expected value E[f(x t-1 , I)].

[0078] In one example, the value of the first moment μ t is [Number] specified as follows.

[0079] The expected value E[f(x t-1 ,I)], that is, the change in the first moment μ t is specified by a tool as described by Anqi Wu, in relation to the edge ε from the context variable I and the distribution of the latent state x t-1 at the previous time point.

[0080] The edge is, in this example, a binary value 0 or 1 in the matrix N m , and these binary values indicate, for example, that an edge exists between two nodes by a value of 1, and that no edge exists between two nodes by a value of 0. The latent state of agent m at time point t [Number] is represented by a node in the graph.

[0081] By the operation AGG, in this example, the message [Number] is specified in relation to the matrix N m and the latent state x t-1 at the previous time point. The message [Number] is concatenated with the latent state x t-1 at the previous time point. The expected value E[f(x t-1 ,I)] is specified by a tool.

[0082] Figure 5b shows a neural network f(x) with layers for operation AGG. t-1 An example of ,I) is shown, followed by 24 fully connected layers and ReLu activation, followed by one fully connected layer and ReLu activation, followed by one more fully connected layer.

[0083] In step 406, for multiple V components v and multiple time points t=1,...T up to the predicted time point T, a normal distribution N(x) is applied to each component v. t |μ t (I),Σ t (I)) the second moment Σ t The value is determined.

[0084] Second moment Σ t The value of is determined recursively in this example. This is the second moment Σ with respect to time t. t The value of is the second moment Σ for a time point preceding time t, for example, t-1. t-1 This means it is identified in relation to the value of .

[0085] In this example, the second moment Σ t The value of is a deterministic change f(x t-1 Covariance Cov[f(x) t-1 ,I) and the second moment Σ t The probabilistic change of L(x) t-1 Expected value E[LL] for ,I) T (x t-1 It is identified in relation to (I).

[0086] The second moment Σ with respect to time t t The value of is the second moment Σ for a preceding time, for example, t-1. t The value and the deterministic change f(x t-1 Covariance Cov[f(x) t-1 ,I) and deterministic change f(x t-1 ,I) preceding time point t t-1 Latent state x t-1 Covariance Cov[xt-1 ,f(x t-1 ,I) and deterministic change f(x t-1 ,I) a preceding point in time, for example t t-1 Latent state x t-1 Covariance Cov[x t-1 ,f(x t-1 The transpose of ,I) and the probabilistic change L(x t-1 Expected value E[LL] for ,I) T (x t-1 It is assumed that it will be identified in relation to (I).

number

[0087] Expected value E[LL] T (x t-1 ,I), that is, the second moment Σ t In one example, the change involves an edge ε from the context variable I and a latent state x at a preceding time point. t-1 Identified in relation to. In this example, identification is performed using the tools E[L] and Cov[L], and therefore E[LL] T (x t-1 ,I)=Cov[L]+E[L]E[L] T That is the case.

[0088] In this example, the edge is matrix N m In this case, the binary value is 0 or 1, and these binary values ​​indicate, for example, that a value of 1 indicates that an edge exists between the two nodes, and a value of 0 indicates that no edge exists between the two nodes. The latent state of agent m at time t.

number

[0089] Operation AGG, in this example, messages

number

number

[0090] Figure 5c shows the neural network L(x) with layers for Operation AGG. t-1 An example of ,I) is shown, followed by 24 fully connected layers and ReLu activation, followed by one fully connected layer and ReLu activation.

[0091] The inner loop includes steps 404 and 406.

[0092] In step 408, for each component v, the normal distribution N(y) at the predicted time T is calculated. t |g(x t ), QQ T (x t The first moment g(x) T,v ) Expected value E[g(x T,v )] is identified. y t The distribution of is, in one example, a Gaussian mixture model called GMMy T ~Σ v π(I)N(y T |a T,v (I), B T,v It is approximated by (I)).

[0093] In this example, for each component v, the first moment g(x) at the predicted time T is... T,v In relation to the value of ), the first moment g(x T,v Covariance Cov[g(x) T,v )] is identified.

[0094] In step 410, the first moment μ at the predicted time T is T,v In relation to the value, the normal distribution N(y t,v |g(x t,v ), QQ T (x t,v The first moment g(x) T,v ) Expected value E[g(x T,v )] is identified.

[0095] In step 412, the normal distribution N(a T,v (I), B T,v (I)) the first moment a T.v (I) is identified.

[0096] In this example, for each component v, the latent state x at the predicted time T is defined. T,v In relation to this, the normal distribution N(y) at prediction time T t |g(x t ), QQ T (x t )) The second moment QQ T The expected value E[QQ] for (x(t)) T (x T,v ) is x t ~N(x_t|μ t,v ,Σ t,v In relation to this, it is identified by tools such as those described in Anqi Wu.

[0097] In this example, the expected value E[g(x T,v ) is the first moment a T.v (I) For example

number

[0098] Expected value E[g(x T,v In this example, this is identified by this tool.

[0099] Figure 5d shows a neural network g(x) with 24 fully connected layers and ReLU activation. t,vAn example of Q(x) is shown, followed by one fully connected layer. t In this example, a constant is assumed for ), but more complex neural networks are also possible.

[0100] In step 414, for each component v, a normal distribution N(a T,v (I), B T,v (I)) the second moment B T,v (I) is identified.

[0101] In this example, for each component v, the first moment g(x T,v Covariance Cov[g(x) T,v )] and the second moment QQ at the predicted time T T (x T,v ) Expected value E[QQ T (x T,v In relation to the normal distribution N(a T,v (I), B T,v (I)) the second moment B T,v (I) For example,

number

[0102] Covariance Cov[g(x T,v ) and expected value E[QQ T (x T,v )] is identified by this tool in this example.

[0103] The outer loop includes steps 408 to 414.

[0104] In step 416, in particular, the third normal distribution N(a) of component v T,v (I), B T,v (I)) at least one weight π v (I) Weighted total

number

[0105] In step 420, behavior

number

number

number

[0106] In step 422, a prediction is output and / or at least one agent 102 is driven and controlled in relation to the prediction.

[0107] For example, computer-controlled machinery, robots, vehicles, household appliances, driven machines, manufacturing machines, personal assistants, or access control systems are driven and controlled.

[0108] For example, predictions for molecular dynamics are identified and output. For example, predictions for movements in sports are identified and output.

[0109] The covariance for various combinations of latent states is given by Dimension MD. x ×MD x It can be treated as a matrix, which contains blocks defined by each of several covariances. In some examples, the matrix is ​​assumed to be approximated as a sparse matrix.

[0110] Figure 6 shows a schematic representation of the approximation of the covariance matrix for the five agents A, B, C, D, and E.

[0111] The latent state of agent m may, in some examples, include multiple elements. In the case of an orbit, the latent state may include, for example, an element relating to agent m's velocity and an element relating to agent m's acceleration. These elements do not necessarily have to be physical quantities and may relate to other aspects of the agent's state.

[0112] In the first approximation of the matrix, only elements from the matrix located on the principal diagonal are used for prediction, and the other elements of the matrix are left unconsidered. This means that multiple latent states of one agent m are modeled independently of each other, and multiple latent states of different agents m are also modeled independently of each other. For example, the velocity of agent m is modeled independently of the acceleration of agent m, and the velocities of different agents m, as well as the accelerations of different agents m, are also modeled independently of each other.

[0113] In Figure 6, the main diagonal is shown as a solid line.

[0114] In the second approximation, multiple latent states of a single agent are modeled independently, while corresponding elements of multiple latent states of different agents are modeled in relation to each other. For example, the velocity and acceleration of the same agent are modeled independently, the velocities of different agents are modeled in relation to each other, and the accelerations of different agents are modeled in relation to each other. This is shown in Figure 6 by the diagonal lines of solid and dashed lines.

[0115] In the third approximation, different elements of one latent state of one agent are modeled in relation to each other, while multiple latent states of different agents are modeled independently of each other. This is shown in Figure 6 by the diagonal lines of the hatched blocks.

[0116] Parameters for parameterizing a neural network: θ = {θ f ,θL} and Ψ={Ψ g ,Ψ Q In this example,} is identified by minimizing the expected negative log probability in the observed trajectory during training with the dataset D={Y,I}:

number

Claims

1. 1. A computer-implemented method for predicting the behavior of an agent (102) in a dynamic system (104) comprising multiple interacting agents (102) with respect to a latent state of the agent (102), comprising: For a plurality of components and a plurality of time points up to a prediction time point, for each component, identifying (404) a value of a first moment of a first distribution that models the latent state of the agent (102) and identifying (406) a value of a second moment of the first distribution; for each component, determining (408) an expectation for a first moment of a second distribution up to the prediction time associated with the value of the first moment of the first distribution up to the prediction time and associated with the value of the second moment of the first distribution up to the prediction time, the second distribution models the behavior of the agent (102) associated with a latent state of the agent (102), and the expectation for the first moment of the second distribution defines (412) a first moment of a third distribution; for each component, determining (414) a second moment of said third distribution, in particular determining (416) a sum of said third distributions for said components weighted by at least one weight; identifying (420) the prediction of the behavior in relation to the aggregate; A method characterized by:

2. determining 404 the value of the first moment of the first distribution relative to values ​​of the first moment of the first distribution for time points preceding the time point and an expectation for a deterministic change in the first moment of the first distribution; and / or 2. The method of claim 1, further comprising: determining (406) the value of the second moment of the first distribution for the time point in relation to values ​​of the second moment of the first distribution for time points preceding the time point in relation to a covariance of the deterministic change and an expectation for a stochastic change of the second moment of the first distribution.

3. 3. The method of claim 2, further comprising: determining (406) the value of the second moment of the first distribution for the time point in relation to the value of the second moment of the first distribution for the preceding time point, the covariance of the deterministic change, the covariance of the latent state for the preceding time point with the deterministic change, the transpose of the covariance of the latent state for the preceding time point with the deterministic change, and the expected value for the stochastic change.

4. The method of claim 1 , further comprising determining (410) the expected value for the first moment of the second distribution relative to the value of the first moment of the first distribution for the prediction time point.

5. For each component, determining 408 a covariance of the first moment of the second distribution relative to the value of the first moment of the second distribution for the prediction time point; For each component, determine 412 an expectation for the second moment of the second distribution for the prediction time associated with a latent state for the prediction time; 2. The method of claim 1, further comprising: determining (416) for each component the second moment of the third distribution relative to the covariance of the first moment of the second distribution and the expected value for the second moment of the second distribution for the forecast time.

6. identifying (402) context variables, the context variables including assignments that assign to at least one agent (102) another agent (102) to be considered for predicting the behavior of the agent, and / or the context variables characterizing a history of the dynamic system (104); 2. The method of claim 1, further comprising: determining the first moment of the first distribution with respect to the context variables (404); and / or determining the second moment of the first distribution with respect to the context variables (406); and / or determining the expected value for the first moment with respect to the context variables (410); and / or determining the first moment of the third distribution with respect to the context variables (414); and / or determining the second moment of the third distribution with respect to the context variables (416); and / or determining, for at least one component, the at least one weight with respect to the context variables (418).

7. identifying (102) the history in relation to observed behavior of the at least one agent (102), particularly in relation to behavior including the position or movement of the agent (102); 7. The method of claim 6, wherein the position or movement of the agent (102) is detected, in particular by a receiver for a satellite-supported positioning system, or at least one digital image is detected, in particular by a sensor for digital images, preferably a camera, a LiDAR sensor, an ultrasonic sensor, a motion sensor, an infrared imaging sensor and / or a radar sensor, and the position or movement of the agent (102) is determined in relation to the at least one digital image, or a signal is detected by a speaker that receives audible sound, and the position or movement of the agent (102) is determined in relation to the signal.

8. the context variables include a matrix, the rows of the matrix representing a respective one of the agents (102), and the columns of the matrix representing a respective one of the agents (102); Identifying (402) a value, particularly a binary value, of at least one of the elements identified by the rows and columns of said matrix, said value setting whether said agent (102) identified by said row should be considered for prediction for said agent (102) identified by said column; or 7. The method of claim 6, further comprising identifying (402) a value, particularly a binary value, of at least one of the elements identified by the rows and columns of the matrix, said value setting whether the agent (102) identified by the column should be considered for predictions for the agent (102) identified by the row.

9. iteratively determining the first moment and the second moment of the first distribution; 2. The method of claim 1, further comprising: determining (402) for each component, for a first one of the iterations, a value of the first moment of the first distribution and a value of the second moment of the first distribution that are associated with the context variable.

10. modeling multiple latent states of one agent (102) without correlation with each other, and modeling multiple latent states of different agents (102) without correlation with each other for the prediction; or Modeling multiple latent states of one agent (102) independently and modeling corresponding elements of multiple latent states of different agents (102) as related to each other; or 2. The method of claim 1, wherein different elements of one latent state of one agent (102) are modeled in a correlated manner, and multiple latent states of different agents (102) are modeled independently.

11. 2. The method according to claim 1, wherein in connection with said prediction, at least one agent (102), in particular a computer-controlled machine, in particular a robot, preferably a vehicle, a household appliance, a driven machine, a manufacturing machine, a personal assistant or an access control system, is driven and controlled (422).

12. The method of claim 1 , wherein the at least one agent (102) is a real object existing in the physical world.

13. An apparatus (100), comprising: The apparatus (100) includes at least one processor (106) and at least one memory (108), the at least one processor (106) and the at least one memory (108) configured to implement the method of claim 1.

1. An apparatus (100) comprising:

14. A system (104), comprising: said system (104) comprising at least one agent (102), in particular a computer-controlled machine, in particular a robot, preferably a vehicle, a household appliance, a driven machine, a manufacturing machine, a personal assistant or an access control system; The agent (102) or the system (104) comprises the device (100) of claim 13, The device (100) is configured to drive and control the agent (102) in relation to a prediction. A system (104).

15. A computer program comprising: The computer program comprises computer readable instructions which, when executed by a computer, perform the method of any one of claims 1 to 12. A computer program characterized by:

Images