using the trained decoder model to make time series predictions

Abstract

The invention relates to a computer-implemented prediction method (900) for making time series predictions for controlling and / or monitoring a computer-controlled system, such as a semi-autonomous vehicle. The method uses a time series of one or more observed states. A state comprises values of measurable quantities of a plurality of interacting objects. Based on the observed states, values of time-invariant latent features of the plurality of objects are determined, e.g. according to an encoder model. Then, at least one next state is predicted using a decoder model. This involves applying a trained graph model to obtain a first prediction contribution based on interactions of an object with other objects, and applying a trained function to obtain a second prediction contribution based on information about the object itself only. Based on the predicted next state, output data is generated for controlling and / or monitoring the computer-controlled system.

Description

Technical Field

[0001] This invention relates to a computer-implemented prediction method that uses a decoder model to perform time series predictions related to computer-controlled systems, such as (semi-)autonomous vehicles. The invention further relates to a computer-implemented training method for training such a decoder model, and to a computer-implemented analysis method for analyzing observations related to a computer-controlled system using an encoder model corresponding to the decoder model. The invention further relates to systems corresponding to these computer-implemented methods, and to a computer-readable medium. Background Technology

[0002] In (semi)autonomous driving systems, a crucial task is analyzing and predicting the environment surrounding the vehicle, such as to automatically brake when a hazardous traffic situation is identified. This specifically involves predicting the trajectories of other road users in the vehicle's environment (e.g., other vehicles, pedestrians, cyclists, etc.) based on their observed previous trajectories. However, in practice, this predictive task has proven complex to perform.

[0003] Similar predictive tasks involving multiple objects interacting with each other over time have also been found in a variety of other application areas, such as robotics (where the objects can be components of a robot, such as parts of a robotic arm, or objects in the environment in which the robot operates, such as other robots or humans); computer networking (where the objects can be, for example, different servers providing related services); power system management (where the objects can be, for example, various components of a power plant); manufacturing (where the objects can be, for example, different manufacturing tools operating in concert, or components assembled together); and epidemics (where the objects can be, for example, an individual disease carrier, or an area in which the disease spreads).

[0004] For many other complex tasks involving real-world data, machine learning is a promising technique for solving prediction problems. Using machine learning, models can be trained to make time-series predictions of the trajectories of interacting objects based on previous observations. In fact, this data-driven approach has proven far more feasible than manually designing prediction algorithms (e.g., explicit systems based on physical equations). Nevertheless, obtaining a prediction algorithm that is accurate enough for real-world applications remains extremely challenging.

[0005] In T. Kipf et al.'s "Neural Relational Inference for Interacting Systems" (available at https: / / arxiv.org / abs / 1802.046877 and incorporated herein by reference), an unsupervised model is proposed that learns to infer interactions in an interactive system while simultaneously learning the system's dynamics purely from observational data. The model consists of a variational autoencoder composed of two jointly trained parts: an encoder that derives interactions given a trajectory, and a decoder that predicts a trajectory given an interaction. The interaction is represented by an interaction graph with nodes representing interacting objects. The encoder learns a discrete edge type for each edge, representing the type of interaction between the two objects connected by that edge. The decoder uses these edge types to predict trajectories using a graph neural network. The model is trained to simultaneously learn to predict edge types and learn dynamic prediction models in an unsupervised manner. Summary of the Invention

[0006] As the inventors recognize, unfortunately, existing methods for predicting the trajectories of interacting objects have many problems. Generally, in many practical situations, the predictions provided by the prior art are not accurate enough. Thus, for example, dangerous traffic situations may still go undetected, or, on the other hand, normal traffic situations may be unreasonably detected as dangerous. The inventors recognize that one reason for this is the failure to adequately consider the global or environmental impact on the prediction mechanism, such as mechanisms that apply individually to objects rather than relying on relationships between them.

[0007] The relevant problem is that existing technologies allow the extraction of information about the relationships between interacting objects, but not about the objects themselves. For example, such information about the objects themselves would be useful for various downstream tasks, such as those in autonomous vehicles that require information about nearby traffic participants.

[0008] Another issue is that existing techniques are often inefficient for training, for example, in terms of computational complexity and / or the amount of data required. This is especially true when training models on real-world observations, such data acquisition is often expensive. Furthermore, given the complexity of the training task, relatively large models are typically required, making training complexity particularly important.

[0009] Therefore, there is a need for a time series prediction technique that provides higher accuracy, allows for the derivation of object-level knowledge, and / or has improved training efficiency.

[0010] According to a first aspect of the present invention, as defined in claims 1 and 12, a computer-implemented method and system for performing time series prediction on multiple interactive objects using a trained decoder model are described. According to another aspect of the present invention, as defined in claims 6 and 13, a computer-implemented training method and system for training such a decoder model are described. According to another aspect of the present invention, as defined in claims 10 and 14, a computer-implemented analysis method and system for analyzing observations of multiple interactive objects using an encoder model corresponding to such a decoder model are described. According to one aspect of the present invention, as defined in claim 15, a computer-readable medium is described.

[0011] Various characteristics relate to making time-series predictions for computer-controlled systems such as robots, (semi-)autonomous vehicles, manufacturing machines, personal assistants, or access control systems. Predictions can involve multiple objects interacting with each other. These multiple objects can include one or more components of the computer-controlled system itself, and / or one or more objects in the environment of the computer-controlled system that enable control and / or monitoring of the system. For example, a vehicle can be controlled based on information about other nearby traffic participants, and possibly based on the vehicle itself. For robots, objects can be, for example, components of the robot itself (e.g., different components of a robotic arm connected to each other by joints), and / or objects in the robot's environment. In power system management, objects can be components of a power plant. In manufacturing, objects can be different jointly operating manufacturing tools, or components assembled together. Therefore, objects are often different physical entities (e.g., independent entities or parts of one or more physical entities), although this is not strictly necessary. For example, objects can be different computer servers, or processes or virtual machines running on computer servers. In an epidemic, objects can be, for example, individual disease carriers, or areas in which the disease spreads.

[0012] Therefore, the decoder model can be used to make time series predictions based on the states of the multiple interacting objects. These states can include the values ​​of one or more measurable quantities for the multiple interacting objects; for example, the state can provide the value of each measurable quantity for each interacting object. Measurable quantities are also called observable quantities. In most cases, they are physical quantities, such as quantities that can be expressed in SI units or combinations of SI units. In typical examples, measurable quantities can include, for example, one or more of the object's position, velocity, and acceleration. Other relevant physical quantities can be the charge of a particle, magnetic flux, forces exerted by or exerted on an object, temperature, etc. Other technical objects, such as processes or virtual machines, can be represented by quantities such as load percentage and / or the amount of resources consumed. In an epidemic, for example, the infection rate can be measured.

[0013] Interestingly, in this context of analyzing and predicting time-series data of interactive objects involving decoder models, the inventors envision using one or more time-invariant latent features to represent the properties of various objects. When making a prediction, these features can be determined from one or more observed states; the decoder model can then use the determined values ​​to predict one or more subsequent states. Here, "time-invariant" means that the model assumes the values ​​of the latent features are constant at least during the considered time window, such as the time window during which observations are available and predictions are made.

[0014] Therefore, a state may only contain those properties of an object that can be directly or indirectly measured (e.g., one or more of position, orientation, attitude, and velocity); time-invariant implicit features can represent other properties that, while not easily or explicitly measurable, may be important for understanding interactions between objects. The inventors recognize that in many physical systems, there exist properties of individual objects (e.g., volume charge governing electrostatic interactions) that help explain their interactions, but which are too expensive or simply impossible to measure. By estimating them from observed states, they can still be used, for example, for prediction.

[0015] As a concrete example, in a traffic scenario, latent features of traffic participants can represent their goals, such as short-term objectives like desired lanes. Latent features can also represent navigation or driving styles, membership in a group of traffic participants, etc. Latent features do not necessarily have predefined meanings; for example, they can be learned in an unsupervised manner based on their applicability for accurate prediction.

[0016] Interestingly, to predict the value of a measurable quantity of an object in its next state, the decoder model can include two distinct components: a trained graphical model and a trained function, also referred to herein as a "local function." The graphical model can account for interactions between the object and other objects, while the local function may not. Both can utilize determined time-invariant latent features. The predictive contributions of the graphical model and the local function can then be combined to obtain a predicted value for the object's measurable quantity.

[0017] As the inventors recognize, by using these two distinct components, the decoder model is effectively forced during training to disentangle predictions concerning interactions from predictions concerning general properties of objects and / or the environment in which they interact. As a result, a model capable of handling well the variability in such properties and / or environmental conditions present in the training data can be obtained. Therefore, the model may be better able to learn and account for global mechanisms that are independent of relationships between objects but apply individually to each object. The decoder model also generally becomes easier to train because two separate models with inherently different inductive biases that reflect the modeling objectives of the components are used.

[0018] Therefore, the following decoder model was obtained: In many cases, this decoder model can provide more accurate predictions and generalize better to unseen situations. These advantages are particularly important in contexts such as autonomous driving, which involves a combination of location-dependent characteristics (e.g., road size, traffic volume, level of compliance with traffic rules) and interaction-related characteristics that may or may not be location-dependent (e.g., how different traffic participants react to each other).

[0019] Specifically, applying graphical models to determine the predicted contribution of a given object can involve determining the pairwise contribution of that object to multiple other objects based on a trained function. This function can take previous values ​​of one or more measurable quantities of the object and the other objects as input, and in some cases (but not necessarily) also the values ​​of time-invariant latent features of the object and the other objects as input. The pairwise contributions can then be aggregated into a total contribution, for example, by summing them or using another type of permutation covariant operator. The advantage of graphical models based on trainable pairwise functions (such as graph neural networks) is that the same graphical model can be used regardless of the number of interacting objects, allowing for better data sharing between different observation scenarios in this regard.

[0020] A graphical model can be a Markov graphical model in the sense that, in order to predict the next state, only a measurable quantity from a previous state is used (but time-invariant latent features are still typically derived based on multiple previous states). Therefore, in this case, the trainable function that determines the pairwise contributions will only take one or more measurable quantities from that previous state as input values. However, it is also possible to use multiple previous states, in which case the pairwise contributions can be determined, for example, based on a recursive model.

[0021] On the other hand, a local function can determine a second predictive contribution based solely on previous values ​​of one or more measurable quantities of the object itself, and possibly also on the values ​​of time-invariant latent features of the object itself. For example, the local function does not use measurable quantities or time-invariant latent features of other objects (although it can use its own time-invariant latent features, which are still typically derived based on measurable quantities of other objects). The local function also does not use time-invariant latent features of other objects. For example, the local function does not use any pairwise contributions with other objects. Therefore, the function can be effectively forced to consider aspects independent of interactions with specific other objects. However, the function can have other inputs.

[0022] For example, the input to a local function can at most include previous values ​​of the object itself, or at most previous values ​​of the object itself and time-invariant hidden features. Similar to graphical models, local functions can also be Markov models, meaning they use the values ​​of measurable quantities from only one previous state, or local functions can be non-Markov models, meaning multiple previous states can be considered, for example, by using a recursive model as a local function.

[0023] The first predictive contribution from the graphical model and the second predictive contribution from the local function can then be combined to obtain predicted values ​​for one or more measurable quantities of the object. Preferably, a predetermined combination function is used, such as a combination function selected based on domain knowledge rather than a combination function trained from the data. It is also preferred to use a feature-by-feature combination function, whereby the first and second predictive contributions may have the same number of elements as the set of measurable quantities, and predictions for the measurable quantities can be obtained by combining the corresponding elements of the first and second predictive contributions. Using a predetermined combination function and / or a feature-by-feature combination function further helps to force the model to deentangle the different aspects of the prediction.

[0024] The most appropriate way to combine predicted contributions often depends on the specific quantity being measured. In many cases, especially with physical quantities, predicted contributions can be combined using mathematical operators such as addition and multiplication. Addition proves to be a good choice in many practical situations. For example, addition is the natural case where the interaction is the result of the sum of physical forces applied to the object. The combination function can directly output the predicted value of the measurable quantity, but its output can also represent the predicted difference from the previous value of the measurable quantity. In this case, the predicted value can be obtained by applying the predicted difference to the previous value, for example, by adding the previous value to the predicted difference.

[0025] To determine the values ​​of time-invariant latent features from observed states, a trained encoder model is used. The encoder model can correspond to the decoder model in the sense that it is trained on top of or together with the decoder model. Typically, training the decoder model is referred to as "forward modeling," while training the encoder model is referred to as "inverse modeling." Therefore, the encoder and decoder can be used jointly to predict the trajectory of an interacting object. The encoder can estimate the latent features given past trajectories. The decoder can receive these inferred latent variables, along with the object's past states, as input and predict their future trajectories. In other words, the decoder can be a generative model of the interacting system and thus can effectively provide a forward mapping from latent variables and past observations to future observations, while the encoder can learn an inverse model mapping from observations to latent variables.

[0026] Similar to decoder models, encoder models can also utilize graphical models, such as graph neural networks. To determine the time-invariant latent features of an object, a graphical model can use the pairwise contributions of that object with multiple other objects. These pairwise contributions can be determined based on the values ​​of one or more measurable quantities of the object and the other objects included in one or more observation states. In encoder models, multiple previous states are typically used; for example, pairwise contributions can be determined on a per-previous-state basis, or on multiple previous states.

[0027] Therefore, the combination of decoder and encoder models can be beneficially used to solve the problem of trajectory prediction and latent feature estimation in scenarios with multiple interacting objects. The time-invariant latent features determined by the encoder model are often features of the objects that govern their behavior, but may be difficult or even impossible to measure directly. Therefore, these latent features can be used not only to make better predictions but also for other downstream tasks. For example, traffic participants can be classified into specific types based on time-invariant latent features; this classification can then be used not only to predict future trajectories more accurately but also in systems that provide users with a visualization of traffic conditions or classify traffic conditions into specific categories (e.g., busy or not busy, dangerous or not dangerous). For example, a traffic condition can be classified as dangerous based on the presence of a large number of children. More generally, additional machine learning models can be applied based on the values ​​of time-invariant latent features. These additional machine learning models can also use observational data. However, in various embodiments, these additional machine learning models do not directly use observational data (e.g., values ​​of measurable quantities for some or all agents) but instead use the values ​​of time-invariant latent features as a proxy.

[0028] Interestingly, the tasks of learning to predict future states and learning to estimate the values ​​of time-invariant latent features can be viewed as two interdependent tasks. On the one hand, recovering informative time-invariant latent feature values ​​improves prediction accuracy. On the other hand, training the encoder model to determine the time-invariant latent features that give good predictions improves the informativeness of the derived latent features. Therefore, using time-invariant latent features can allow for improvements in both predictions made by the decoder model and the analysis of time-series observations by the encoder model.

[0029] Regarding the training of the decoder model, one possibility is to perform regular end-to-end training of the decoder model together with the encoder model. That is, based on observed scenes, the decoder model can be trained by selecting a scene; the encoder model can be applied to determine the values ​​of the time-invariant latent features of objects in that scene; the decoder model can be applied to predict states also included in that scene; and this prediction can be compared with the actual state. Based on this comparison, a training signal can be derived, and based on this training signal, the parameters of both the encoder and decoder models can be updated.

[0030] Performing conventional end-to-end training may seem to be the only possibility, as the tasks of dynamic model learning and hidden feature estimation appear to be interdependent. Training the decoder model will seem to require sufficiently good estimates of the latent variables, while inference of the latent variables will seem to require the availability of an accurate forward (i.e., decoder) model.

[0031] However, interestingly, the inventors recognized that it is also possible to separate the training of the encoder model and the decoder model. That is, when training the decoder model, the values ​​of the time-invariant latent features of the objects in the scene can be jointly learned as trainable parameters along with the trainable parameters of the decoder model. Based on these learned values ​​of the time-invariant latent features, the encoder model can then be learned in a supervised manner.

[0032] Using this two-step approach to perform training offers several advantages. Training both the encoder and decoder simultaneously is a complex learning task that typically exhibits a slow convergence rate. By instead splitting the training into two distinct and easier learning tasks, the computational complexity of the training is improved, potentially requiring fewer iterations to reach an accurate model. Furthermore, the risk of reaching local optima is reduced, and thus, in practice, higher accuracy has been observed in the resulting encoder and decoder models, resulting in more accurate predictions. Training in the two-step approach also works particularly well with the time-invariant latent features of the objects. For example, compared to pairwise latent features used in the art, a significantly smaller number of latent features are required as trainable parameters (e.g., linear rather than quadratic in terms of the number of objects), making them easier to train.

[0033] Interestingly, the two-step training approach also allows for semi-supervised learning. That is, latent feature values ​​can be used as training data for a subset of objects and / or time-invariant latent features and / or scenes. For example, latent features can be manually annotated for objects across multiple scenes. In practice, this might be too expensive for the entire dataset, but it is feasible for a subset of the data (e.g., at most 5% or 10%). These known latent feature values ​​can then be fixed instead of being used as trainable parameters when training the decision-maker model. Interestingly, performing semi-supervised joint training of the encoder and decoder not only improves the performance of the decoder for prediction but also improves the performance of the encoder for deriving latent feature values, regardless of whether the encoder is used in conjunction with the decoder.

[0034] Another interesting aspect of using time-invariant latent features as trainable parameters to train the decoder model is the possibility of training the decoder model without explicitly training a separate trainable encoder model. For example, as discussed elsewhere, it is possible to use the decoder model instead to derive the time-invariant latent features from the observed states, for instance, by performing optimization to determine the values ​​of the time-invariant latent features based on which the decoder model can best predict a given observed state. Similarly, time-invariant latent features can be determined in this way when performing time series predictions as described in this paper, without using a trainable encoder model separate from the decoder model. Nevertheless, the use of an encoder model is generally still preferred, for example, to allow for more efficient computation of latent features.

[0035] Optionally, some or all of the time-invariant latent features can be discrete features of a corresponding finite set of possible values. Therefore, determining the latent features of an object may essentially correspond to classifying the object into categories corresponding to the possible values. When using a decoder model to determine the time-invariant latent features for prediction, the output of the trained encoder model can then be discretized into a finite set. When training the decoder, the training signal can be configured to penalize the deviation of the time-invariant latent features (e.g., given as trainable parameters, or as the encoder output when performing end-to-end training) from the finite set of possible values.

[0036] Generally, classification learning problems that involve selecting a value from a finite set of possible values ​​are easier to solve than regression problems, especially in an unsupervised context. Therefore, the inventors have found that the use of discrete features significantly improves learning performance and produces higher-quality predictions, and is therefore preferred, especially when combined with unsupervised learning, in which no latent feature values ​​are given as training data. Moreover, using discrete object-level features is advantageous compared to using discrete pairwise features as done in the prior art. In many cases, using object-level features is a much simpler modeling choice. The number of features is smaller, for example, linear rather than quadratic in terms of the number of objects. Another reason is that combinatorial explosion of the latent space can be avoided, since the number of object pairs of related types is usually expected to scale quadratically with the number of objects of the related type itself. Another reason is that, for example, instead of using corresponding models for different interaction types, model parameters can be shared across interaction types. Therefore, data efficiency is improved.

[0037] Generally, predicted states and / or determined time-invariant features, as described herein, can be used in a manner otherwise conventional when controlling and / or monitoring computer-controlled systems at hand. For example, states and / or features can be provided to control units of, for example, semi-autonomous vehicles, robots, or other types of computer-controlled systems, to achieve appropriate actions. Numerous examples exist where predicted trajectories are required for such control and monitoring tasks, and the proposed techniques can be readily combined with these tasks. In particular, additional machine learning models, such as models classifying traffic situations as hazardous or safe, can be applied to determine the output data for control and / or monitoring, for example, to initiate braking in hazardous traffic situations. As another example, (semi-)autonomous vehicles can use predicted trajectories of other traffic participants to plan safe navigation paths toward a target, for example, safe navigation paths that minimize collision risk.

[0038] Optionally, the output data can be provided to an output device via an output interface, which is used when controlling or monitoring a computer-controlled system. This output device can be an actuator associated with the system, which controls the system by providing control data to the actuator based on predicted (one or more) next states or determined latent characteristics of the system. For example, the actuator can be used to control vehicles such as autonomous or semi-autonomous vehicles, robots, manufacturing machines, buildings, etc.

[0039] Those skilled in the art will appreciate that two or more of the above embodiments, implementations and / or optional aspects of the present invention can be combined in any manner deemed useful.

[0040] Those skilled in the art can make modifications and variations to any system and / or any computer-readable medium based on this description, corresponding to the modifications and variations described in the description of the corresponding computer implementation method. Attached Figure Description

[0041] These and other aspects of the invention will become apparent and further illustrated by reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, in which: Figure 1 A system for training a decoder model is shown, which is used for time series prediction of multiple interacting objects; Figure 2 A system for performing time series forecasting or analyzing observations of multiple interacting objects is shown; Figure 3 A (semi-)autonomous vehicle operating in an environment using a system as described herein is shown; Figure 4 A (semi-)autonomous vehicle operating in the environment is shown; Figure 5 A detailed example of a graph model is shown; Figure 6 A detailed example of how to predict at least one next state is shown; Figure 7a A detailed example of determining pairwise contributions used in the encoder model is shown; Figure 7b A detailed example of determining pairwise contributions used in the encoder model is shown; Figure 8a A detailed example of how to train a decoder model is shown; Figure 8b A detailed example of how to train an encoder model is shown; Figure 8c A detailed example of how to train an encoder model is shown; Figure 9 A computer implementation method for time series forecasting of multiple interactive objects is shown; Figure 10 A computer implementation method for training the decoder model is shown; Figure 11 A computer implementation method for analyzing observations from multiple interacting objects is shown; Figure 12 A computer-readable medium containing data is shown.

[0042] It should be noted that the accompanying drawings are purely schematic and not drawn to scale. In the drawings, elements corresponding to those already described may have the same reference numerals. Detailed Implementation

[0043] Figure 1 A training system 100 is illustrated for training a decoder model 041 for time-series prediction of multiple interacting objects in a computer-controlled system. The corresponding objects may be part of the computer-controlled system or its environment. System 100 may include a data interface 120 for accessing training data 030 representing multiple observed scenes. As shown in the figure, data interface 120 may also be used to access model data 041 representing the decoder model, and optionally to access model data 042 representing an encoder model, which is also trained by system 100. Model data 041, 042 may be used according to the methods described herein, for example, via Figure 2System 200 is used to analyze observations of interactive objects and / or perform time series predictions on interactive objects. System 100 can be configured to apply (one or more) models 041, 042 in addition to training them; for example, system 100 can work with... Figure 2 The system has 200 combinations.

[0044] For example, and also Figure 1 As illustrated, the data interface can be comprised of a data storage interface 120, which can access data 030, 041, and 042 from data storage device 021. For example, the data storage interface 120 can be a memory interface or a persistent storage interface (e.g., a hard disk or SSD interface), but it can also be a personal, local area network, or wide area network interface (such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface). The data storage device 021 can be an internal data storage device of system 100 (such as a hard disk or SSD), but it can also be an external data storage device (e.g., a network-accessible data storage device). In some embodiments, data 030, 041, and 042 can all be accessed from different data storage devices, for example, via different subsystems of the data storage interface 120. Each subsystem can belong to the type described above for the data storage interface 120.

[0045] System 100 may further include a processor subsystem 140, which may be configured to train a decoder model during operation of system 100. Training may be performed in multiple iterations. In each iteration, processor subsystem 140 may be configured to select observed scenes from a plurality of observed scenes. The observed scenes may represent time series of multiple observed states of multiple interacting objects that are part of a computer-controlled system or its environment. States may include values ​​of one or more measurable quantities of the plurality of interacting objects. Processor subsystem 140 may further obtain values ​​of one or more time-invariant latent features of the plurality of objects in the iterations and may predict the state of the observed scenes based on previous states of the observed scenes, for example, using time-invariant latent features.

[0046] To make a prediction, the processor subsystem can apply a graphical model of decoder model 041 to obtain a first predictive contribution for the object. The first predictive contribution can be based on pairwise contributions between the object and multiple other objects. The pairwise contributions between the object and other objects can be determined based on the values ​​of time-invariant latent features of the object and other objects, as well as previous values ​​of one or more measurable quantities. To make a prediction, the processor subsystem 140 can further apply a function of the decoder model to obtain a second predictive contribution for the object. The second predictive contribution can be based on previous values ​​of one or more measurable quantities of the object itself, and optionally on the values ​​of time-invariant latent features of the object itself. To make a prediction, the processor subsystem 140 can further combine the first and second predictive contributions to obtain predicted values ​​for one or more measurable quantities of the object.

[0047] As part of the training iteration, the processor subsystem 140 can further derive a training signal based on comparing the observed state of the scene with its prediction, and can adapt the parameters of the graphical model and / or the function based on the training signal.

[0048] System 100 may further include output interfaces for outputting model data 041 and 042, respectively, representing the trained decoder model and encoder model (if used). For example, also... Figure 1 As illustrated, the output interface can be comprised of a data interface 120, which in these embodiments is an input / output (“IO”) interface through which trained model data 041, 042 can be stored in data storage device 021. For example, model data defining an “untrained” model can be at least partially replaced by model data of a trained model during or after training, because the parameters of the dynamic model (such as the weights of the neural network and other types of parameters) can be adapted to reflect training on training data 030. This also... Figure 1 The diagram illustrates this by showing untrained and trained model data as the same record. In other embodiments, trained model data may be stored separately from model data defining an "untrained" model. In some embodiments, the output interface may be separate from the data storage interface 120, but it can generally be of the type described above for the data storage interface 120.

[0049] Figure 2A system 200 is shown for generating output data for controlling and / or monitoring a computer-controlled system. Specifically, the system can be used to perform time-series predictions on multiple interacting objects that are part of the computer-controlled system or its environment based on a decoder model, and / or to analyze observations of such interacting objects based on an encoder model, based on which the output data can be generated.

[0050] System 200 may include a data interface 220 for accessing model data 041 representing a trained decoder model and / or model data 042 representing a trained encoder model. The decoder model and / or encoder model may be generated by... Figure 1 The system is generated by system 100 or other methods described in the system.

[0051] For example, and also Figure 2 As illustrated in the diagram, the data interface can be composed of a data storage interface 220, which can access data 041 and 042 from the data storage device 022. Generally, the data interface 220 and the data storage device 022 can belong to the reference... Figure 1 The same type of data interface 120 and data storage device 021 are described. Data storage device 022 may also include observation data, which represents a time series of one or more observation states of multiple interacting objects. A state may include the value of one or more measurable quantities of the multiple interacting objects. As described elsewhere, in some embodiments, observation data may also be received directly from sensor 072 via sensor interface 260 or via another type of interface, rather than accessed from data storage device 022 via data storage interface 220.

[0052] System 200 may further include processor subsystem 240, which may be configured to acquire observation data during operation of system 200, for example, via storage interface 220 or via sensor interface 260.

[0053] When using observed data for time series forecasting, processor subsystem 240 can be configured to determine the values ​​of one or more time-invariant latent features of a plurality of objects based on one or more observed states. Processor subsystem 240 can then predict at least one next state of the plurality of interacting objects. To make the prediction, processor subsystem 240 can apply a trained graphical model of decoder model 041 to obtain a first predictive contribution for the object. The first predictive contribution can be based on pairwise contributions of the object with multiple other objects. The pairwise contribution of the object with other objects can be determined based on the values ​​of the time-invariant latent features of the object and other objects, as well as previous values ​​of one or more measurable quantities. To make the prediction, processor subsystem 240 can further apply a trained function of decoder model 041 to obtain a second predictive contribution for the object, the second predictive contribution being based on previous values ​​of one or more measurable quantities of the object itself, and optionally on the values ​​of the time-invariant latent features of the object itself. To make the prediction, processor subsystem 240 can combine the first and second predictive contributions to obtain predicted values ​​of one or more measurable quantities of the object. The processor subsystem 240 can then generate output data for control and / or monitoring based on at least one predicted next state.

[0054] When using observational data to analyze observational results, processor subsystem 240 can use encoder model 042 to determine the values ​​of one or more time-invariant latent features of the plurality of objects based on the one or more states. This determination may include applying a trained graphical model of encoder model 042 based on the pairwise contributions of the object to multiple other objects. The pairwise contributions of the object to the other objects can be determined based on the values ​​of one or more measurable quantities of the object and the other objects included in one or more observational states. Processor subsystem 240 can then generate output data for control and / or monitoring based on the values ​​of one or more time-invariant latent features, which may optionally include applying additional machine learning models based on the values ​​of the time-invariant latent features.

[0055] What will be understood is, for example, regarding Figure 1 Like processor subsystem 140, the same considerations and implementation options apply to processor subsystem 240. It will also be further understood that, as for... Figure 1 As with System 100, the same considerations and implementation options can generally be applied to System 200, unless otherwise indicated.

[0056] Figure 2Various optional components of system 200 are further illustrated. For example, in some embodiments, system 200 may include sensor interface 260 for direct access to sensor data 224 acquired by sensor 072 in environment 082. The sensor may be positioned within environment 082, but may also be positioned remotely from environment 082, for example, if one or more quantities can be measured remotely. Sensor 072 may, but does not need to, be part of system 200. Sensor 072 may have any suitable form, such as an image sensor, video sensor, lidar sensor, radar sensor, ultrasonic sensor, motion sensor, thermal imaging camera, pressure sensor, temperature sensor, etc. Observational data can be derived from sensor data 072 using conventional means, for example, the location of an object can be derived from image or video data using image extraction techniques. Sensor data interface 260 may have any suitable form corresponding in type to the sensor type, including but not limited to: low-level communication interfaces such as I2C or SPI-based data communication, or data storage interfaces of the type described above for data interface 220.

[0057] In some embodiments, system 200 may include actuator interface 280 for providing control data 226 to actuators (not shown) in environment 082. This control data 226 may be generated by processor subsystem 240 to control the actuator based on derived values ​​of predicted next states or time-invariant latent features determined using decoder model 041 or encoder model 042. The actuator may be part of system 200. For example, the actuator may be an electric, hydraulic, pneumatic, thermal, magnetic, and / or mechanical actuator. Specific but non-limiting examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servo mechanisms, solenoids, stepper motors, etc. Reference Figure 3 This type of control is described for (semi)autonomous vehicles.

[0058] In other embodiments ( Figure 2 In (not shown), system 200 may include an output interface to a presentation device, such as a display, light source, speaker, vibration motor, etc., which can be used to generate sensory-perceptible output signals based on predicted states or determined latent feature values. Sensory-perceptible output signals can directly indicate these model outputs, but can also represent derived sensory-perceptible output signals, such as those used for guiding, navigating, or other types of control of computer-controlled systems.

[0059] Generally speaking, each system described in this specification (including but not limited to) Figure 1 System 100 and Figure 2System 200 can be implemented as a single device or apparatus (such as a workstation or server) or implemented within a single device or apparatus. The device can be an embedded device. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem of the corresponding system can be implemented by a single central processing unit (CPU), but can also be implemented by a combination or system of such a CPU and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory (such as RAM) or non-volatile memory (such as flash memory). Alternatively, the processor subsystem of the corresponding system can be implemented in the device or apparatus as programmable logic, such as a field-programmable gate array (FPGA). Generally, each functional unit of the corresponding system can be implemented as a circuit. The corresponding system can also be implemented in a distributed manner, such as involving different devices or apparatuses, such as distributed local servers or cloud-based servers. In some embodiments, system 200 can be part of a computer-controlled system (e.g., a vehicle, robot, etc.) and / or can represent a control system configured to control a computer-controlled system.

[0060] Figure 3 An example of the above is shown, in which system 200 is shown as a control system for an autonomous vehicle 62 (in this case, a car) operating in environment 50. The autonomous vehicle 62 can be autonomous, as it may include an autonomous driving system or a driver assistance system, the latter also referred to as a semi-autonomous system.

[0061] In this example, the state may include location information about multiple traffic participants in the environment 50 of the vehicle. For example, the location information may include one or more of the traffic participants' position, velocity, attitude, and acceleration. Typically, the location information in this example is two-dimensional. The subsequent state(s) predicted by system 200 can be used to monitor and / or control the autonomous vehicle in many different ways.

[0062] For example, autonomous vehicle 62 can incorporate system 200 to generate control data for steering and / or braking of the autonomous vehicle based on one or more subsequent states predicted by system 200. The states can be predicted based on observation data derived from a video camera 22 integrated into vehicle 62.

[0063] For example, control data can be generated to control motor 42 to perform (regenerative) braking in the event of a anticipated collision between autonomous vehicle 62 and a traffic participant. Steering and / or braking can be controlled to avoid a collision with a traffic participant. For this purpose, system 200 can predict one or more upcoming states, including the positions of one or more traffic participants, based on sensor data obtained from a video camera. If the future state of a traffic participant is expected to result in a collision, system 200 can take appropriate action.

[0064] Figure 4 A vehicle 62 operating in environment 50 is shown (e.g., Figure 3 A top view of the vehicles 62. For example, multiple buildings 401 and multiple sidewalks 402 are shown. Zebra crossings 414 and traffic lights 412 are also shown in the figure.

[0065] This figure illustrates a traffic situation with multiple traffic participants in the environment 50 of vehicle 62. The state of traffic participants in vehicle 62 and / or its environment, including location information about the multiple traffic participants, can be predicted according to the techniques described herein. In this example, the traffic participants include several pedestrians 420-423. In this non-limiting example, four pedestrians are shown. These pedestrians may operate in environment 50, for example, according to short-term goals, and in a sense that their behavior can be expected to depend on the presence and behavior of other traffic participants, these pedestrians may also interact with each other and / or with vehicle 62.

[0066] For example, it can be expected that the probability of a pedestrian crossing the road increases when the pedestrian approaches zebra crossing 414. Therefore, for example, the short-term goal of the pedestrian can be derived by an encoder model based on the pedestrian's movement toward and / or towards the crosswalk. For example, pedestrians 421 and 422 may be more likely to cross the road than pedestrians 420 and 423.

[0067] Different derived short-term objectives can lead to different predicted trajectories. For example, the future trajectory of pedestrian 421 may depend not only on whether crossing the road is the pedestrian's short-term objective, but also on the short-term objectives of other traffic participants. On the other hand, a pedestrian's short-term objective may influence the trajectories of other traffic participants. For example, if the short-term objective of vehicle 62 is to travel in a straight line and the short-term objective of pedestrian 621 is to cross the road, vehicle 62 may be predicted to slow down and wait for the pedestrian; and / or pedestrian 621 may be predicted to pause to see if the vehicle is actually waiting. However, if the short-term objective of vehicle 62 is to turn left, or the objective of pedestrian 621 is to remain on the crosswalk, their trajectories may not influence each other.

[0068] In this example, the short-term goal of vehicle 62, which includes the prediction / analysis system 200, is used. However, this example also applies to situations where vehicle 62 does not include the prediction / analysis system itself, but instead, vehicle 62 is a vehicle in the environment of the system.

[0069] Interestingly, the model described in this paper allows learning from data how such short-term goals, as well as other information that can be indirectly derived from direct measurements of quantities such as position and acceleration, influence a pedestrian's future trajectory. Furthermore, this information influencing the trajectory captured by the time-invariant latent features of these multiple objects can be automatically derived from the data. Therefore, more accurate predictions of future trajectories can be obtained.

[0070] Figure 5 Detailed, but non-limiting, examples of graph models used in encoder and / or decoder models are shown. Generally, a graph model is a model in which data is modeled according to a graph structure. The characteristics of the nodes in the graph can be determined by determining the pairwise contributions of the node with other nodes in the graph based on a learnable function, and combining these pairwise contributions into an aggregated pairwise contribution for the object. Here, aggregation typically supports any number of other nodes; for example, aggregation may include summing the pairwise contributions together, so that the entire graph model can typically also support any number of objects. The graph model is preferably permutation-covariant, for example, the order of objects in the graph may not affect the predictions made for the corresponding objects. For example, a graph model can be configured such that permuting multiple objects in the input to the graph model results in the same permutation of the predictions output by the model for those multiple objects. This allows the model to better represent sets of objects that do not have an inherent order among them.

[0071] For example, the graph model described in this paper can be a graph neural network, such as that disclosed in F. Scarselli et al., “The graph neural network model,” IEEE Transactions on Neural Networks, 2009 (incorporated hereby by reference). The graph model described in this paper can also be an interaction network, such as that disclosed in P. Battaglia et al., “Interaction Networks for Learning about Objects, Relations and Physics” (incorporated hereby by reference and available at https: / / arxiv.org / abs / 1612.00222).

[0072] This diagram illustrates the specific graph model architecture that can be used for both the encoder and decoder models described in this paper. The graph model in this diagram is a single-layer graph model on a fully connected graph with interactive objects as nodes. The graph model obtains the node-level input feature vector Xi510 for the corresponding interactive object and calculates the node-level output feature vector Oi570.

[0073] Specifically, the figure illustrates the input feature vectors X1...XA510 representing the corresponding interactive objects, whose behavior is modeled by a graph model. The input feature vectors are typically real-valued vectors, for example, having one, two, at most or at least five, or at most or at least ten features. Typically, an object's feature vector includes the values ​​of the object's measurable quantities and / or time-invariant latent features, as also described elsewhere.

[0074] The figure also shows the pairwise contribution function PCF520, also known as the edge update function. The pairwise contribution function can be applied to input feature vector pairs Xi and Xj from different objects. The pairwise contribution function PCF can be, for example, a neural network or any other suitable machine-learnable function. Applying the pairwise contribution function PCF to the input feature vector pair Xi and Xj produces the pairwise contribution PCij530. Mathematically, this can be represented as: in This indicates vector concatenation, and [ A ] = {1, 2, ..., A} is the front A A set of n natural numbers.

[0075] Then, based on the aggregation function AGG540, the pairwise contributions PCij can be aggregated row-by-row or column-by-column for each object, for example, according to the grid structure shown in the figure, thus producing the aggregated pairwise contribution APCi550 for each object. Aggregation functions are typically permutationally covariant; for example, changing the order of the independent variables going to the function will not affect the result. For example, aggregation functions can be addition, multiplication, average, maximum, or similar functions. Aggregation functions are usually predefined; for example, they are not parameterized by any trainable parameters. Mathematically, this can be represented as: Finally, the optional node update function NUF560 can be applied to the corresponding aggregated pairwise contributions to obtain the corresponding output Oi570 of the graph model. The node update function is typically also a machine-learnable function, such as that of a neural network. Mathematically, this can be represented as: Preferably, the functions constituting the graphical model It is differentiable, making it easier to train graphical models, for example, using gradient descent. For instance, the function of a graphical model can be a multilayer perceptron or a recursive model, such as a gated recurrent unit or a long short-term memory network. In the case of an encoder model with discrete-time invariant hidden features, the node update function NUF that determines the output of the encoder model can include functions for constraining the output to values ​​sufficiently close to the desired discrete values, such as softmax, sigmoid, or tanh output layers.

[0076] Although this diagram illustrates a single-layer graph model, the graph models used in this paper can typically include multiple layers, such as two, three, or more. For example, multi-layer graph models can be applied multiple times, such as... Figure 5 The same or different graphical models shown can be used to implement this.

[0077] Figure 6 A detailed, but non-limiting, example of how to predict at least one of the next states of multiple interacting objects is shown. Based on the predicted state(s), output data can be generated for controlling and / or monitoring computer-controlled systems, as also described elsewhere.

[0078] The figure illustrates observation data X1...XA610, representing a time series of one or more observation states. Typically, multiple observation states exist. The time intervals between subsequent observation states are usually constant. For example, the number of observation states used can be at most or at least 10, at most or at least 100, or at most or at least 500. States can include values ​​of one or more measurable quantities of multiple interacting objects of a computer-controlled system and / or its environment. In this figure, the observation data is shown as a time series comprising elements X1...XA, each element representing the value of one or more measurable quantities of the corresponding object. This time series for each object can be referred to as the trajectory of that object. Therefore, a scene represented by the observation data can be viewed as a trajectory comprising multiple objects, the trajectory including measurements of one or more measurable quantities at a common set of time points.

[0079] For example, the number of objects A There can be at most or at least 5, at most or at least 10, or at most or at least 50 features. One or more measurable quantities of the corresponding object at a certain point in time can be represented by a feature vector (e.g., a real-valued feature vector). For example, the feature vector can consist of at least 2, at least 4, at most or at least 6, or at most or at least 10 features.

[0080] Measurable quantities can include information about the location of an object. For example, the location of the object can be included. For instance, a measurable quantity at a given point in time can be represented as 2D or 3D coordinates. Alternatively or additionally, a measurable quantity can include the velocity of the object, for example, in 2D or 3D, expressed as a number or a vector. Similarly, alternatively or additionally, a measurable quantity can include the acceleration of the object (e.g., in 2D or 3D), expressed as a number or a vector. Generally, location observations can be obtained from various modalities using conventional techniques, such as image extraction from camera or video data, radar data, etc.

[0081] However, in addition to, or instead of, location information, other measurable quantities can be used. For example, measurements of temperature, air pressure, etc., can be used. For a robot, a measurable quantity could be, for example, the amount of pressure exerted by or applied to the robot's components. These are examples of physical quantities, although they are not strictly required. For a computer server or virtual machine, a measurable quantity could be load, the number of requests per time interval, etc. Generally, measurable quantities can be measured directly, for example, using appropriate sensors, or indirectly, for example, by deriving the measurement result from the measurement of other quantities.

[0082] Interestingly, time-invariant implicit features can be used to predict the next state of one or more interacting objects. For each of the multiple objects, the values ​​of these time-invariant implicit features can be determined based on the observed state Xi. The figure shows the values ​​H1...HA690 of one or more time-invariant implicit features for the corresponding objects; for example, H1 could represent the value of the time-invariant implicit feature of the first object, H2 could represent the value of the time-invariant implicit feature of the second object, and so on. The number of time-invariant implicit features can be, for example, one, at least two, at most or at least five, or at most or at least ten. It is possible to use continuous time-invariant implicit features, but in a preferred embodiment, some or all of the time-invariant implicit features are discrete. That is, the time-invariant implicit features have values ​​from a finite set of possible values. This number of possible values ​​is usually pre-configured. For example, the number of possible values ​​for a time-invariant implicit feature can be two, three, at most or at least five, or at most or at least ten. One-hot encoding can be used to represent discrete time-invariant implicit features.

[0083] Various options are available for calculating the value of the time-invariant latent feature Hi. The figure illustrates the encoder model Enc601 being used. Encoder models can include trained graphical models, such as those related to... Figure 5As described. Therefore, determining the time-invariant latent feature Hi can include applying the trained graphical model based on the pairwise contributions of the object with multiple other objects. The pairwise contribution of the object with other objects can be determined based on the values ​​Xi, Xj of one or more measurable quantities of the object and other objects included in one or more observation states. (The remaining text appears to be incomplete and requires further context.) Figure 7a and 7b Let's discuss the various particularly advantageous options for the encoder model Enc.

[0084] However, it is not necessary to use a separate encoder model Enc to determine the time-invariant latent features Hi. Instead, the time-invariant latent features Hi can be determined based on the decoder model discussed in the figure, for example, by performing optimization to determine the values ​​Hi that provide good predictions for the observations Xi. For example, given candidate values ​​for the time-invariant latent features and a subset of the states of the observed data Xi, predictions can be made for another subset of the states of the observed data Xi, for example, as described with respect to the figure. For example, at time point 1... T The state at -1 can be used to predict time point 2... T The corresponding state at that point. Then, candidate values ​​of the latent features that yield the best prediction for those known states can also be used to predict the next state for which no observations are available. Using the decoder model to determine the time-invariant latent features will be about Figure 8a and 8c It was described further.

[0085] Whether using an encoder model Enc or otherwise to determine the time-invariant latent feature Hi, discretization can be applied to determine the discrete values ​​of the time-invariant latent feature. For example, a value indicating the degree of correspondence or likelihood of a corresponding discrete value for a time-invariant latent feature can be discretized into a single value, such as an integer or one-hot encoding of the time-invariant latent feature.

[0086] Given observation data Xi and time-invariant latent features Hi, it is possible to predict one or more subsequent states of multiple interacting objects based on a trained decoder model. Mathematically, this can be expressed as follows. Consider... A A collection of objects, the states of which can be defined at discrete time points 1... T The location is detected as the value of one or more measurable quantities. Objects included in the observation data Xi. i In time t The state at can be represented as ,in F It is the feature dimension, for example, the number of features used to represent one or more measurable quantities.i The time-invariant latent feature Hi of an object can be represented as... h i .For example, h i It can be included in K At least one discrete feature is obtained over 10 possible values. Interestingly, latent features may not be directly observable.

[0087] In various embodiments, the decoder model is a Markov model, meaning that based on the observed data Xi, the decoder model only uses the most recent state. For example, the agent can be... i In time t Observation status at the location Modeling as the last observed state of all agents and their hidden features The function. Tensors can be used. x ∈ To represent all A Each object in all T The values ​​of one or more measurable quantities at each time step. For time-invariant implicit features, feature vectors can be used. Or, for example, discrete time-invariant latent features can be represented as tensors. This tensor represents the value of the latent feature of the corresponding object, for example, its one-hot encoding or continuous relaxation.

[0088] As shown in the figure, the decoder model can include a trained graphical model GM602 and a trained local function LF603. The graphical model GM can be applied to obtain a first predictive contribution FPCi670 for the corresponding object, while the local function LF can be used to obtain a second predictive contribution SPCi671 for the corresponding object. Both modules can take observation data Xi and possibly latent features Hi as input. Interestingly, the graphical model GM can be based on pairwise contributions, as discussed further below, thus effectively allowing information about the state of objects to be shared among them; while the local function LF can process the data for each object independently (e.g., observation Xi and / or latent features Hi). Therefore, the graphical model GM can be effectively forced to model the interactions of objects, while the local function LF can be effectively forced to model the behavior exhibited by each object, independent of other things (such as interactions with the environment). Then, in the combination operation CMB680, the first and second contributions FPCi, SPCi can be combined to obtain predicted values ​​xi,t+1611 for one or more measurable quantities.

[0089] The graphical model GM used to obtain the first predictive contribution FPCi can use information about... Figure 5 The architecture discussed. The first predicted contribution FPCi for an object can be based on the pairwise contributions of that object with multiple other objects. The pairwise contribution can be determined based on the values ​​Hi, Hj of the time-invariant latent features of that object and other objects, as well as the previous values ​​Xi, Xj of one or more measurable quantities. The previous values ​​can come from the immediately preceding state, thus obtaining a Markov graphical model, or from multiple previous states.

[0090] As a specific example, when using Figure 5 When designing a graphical model architecture, the following options can be considered. The pairwise contribution function can be a multilayer perceptron, which takes the most recent previous value and time-invariant latent features as input. The pairwise contributions can then be aggregated by summation. After this, a node update function provided by another multilayer perceptron can be applied to obtain the first prediction contribution. In mathematics, this can be represented as: .

[0091] The trained local function LF can determine a second predictive contribution SPCi for an object based on previous values ​​of a measurable quantity Xi. This can be a value from the most recent previous state to obtain the Markov function, or it can be a value from multiple previous states. The trained local function can also optionally use the value of the object's time-invariant latent feature Hi. However, previous values ​​and latent features of other objects are not input into the local function LF. For example, the local function LF can be implemented as a multilayer perceptron, such as: .

[0092] To combine the first and second contributions, FPCi and SPCi, to obtain the predicted value xi,t+1, the combination operation CMB typically applies a predefined combination function, such as a combination function that is not parameterized by trainable parameters. The combination function can also use previous states; for example, combining the contributions can result in an offset applied to the immediately preceding state. The combination function can operate element-wise, such as through element-wise addition. The first and second contributions, as well as the predicted xi,t+1, can all have the same number of elements, corresponding to the number of features used to represent one or more measurable quantities. For example, the prediction... x t+1 and the state immediately preceding it x t They can be related according to the following formula: .

[0093] In some embodiments, multiple subsequent states of the interacting object can be predicted. For this purpose, predicted values ​​xi,t+1 of one or more measurable quantities can (optionally, in combination with previous predictions or observations) be used as input to further applications to the decoder model. Typically, the same values ​​of latent features are reused, while the encoder model Enc is not usually reused. For example, in this way, states can be predicted for at least 2, at most 10, or at most 50 time steps into the future.

[0094] Figure 7a Detailed, but non-limiting, examples of determining pairwise contributions used in encoder models are shown.

[0095] This figure illustrates the pairwise contribution function PCF720, which is used to determine the pairwise contributions pi,j,T730 of a given object to other objects based on the observed data Xi710. The observed data can represent a time series of states, where each state includes the values ​​of one or more measurable quantities from multiple interacting objects. (About...) Figure 6 The observational data were further discussed. According to Figure 5 In this architecture, the pairwise contribution function (PCF) can be used as a pairwise contribution function in the encoder model.

[0096] In this example, the pairwise contribution pi,j,T can be determined by applying a trainable recursive model RM 725 to the values ​​of one or more measurable quantities of the object and another object. Therefore, the pairwise contribution can be determined by continuously updating the value pi,j,t 736 of the pairwise contribution from the previous value pi,j,t-1 of the pairwise contribution according to the recursive model RM. The initial value of the pairwise contribution can be initialized with zero (e.g., pi,j,0). The recursive model RM can then be applied iteratively, for example, starting with pi,j,0 to obtain pi,j,1, pi,j,2, etc. The pairwise contribution pi,j,t for the most recent state can then be used as the output pi,j,T of the pairwise contribution function. Known recursive models (e.g., recursive neural networks, such as Long Short-Term Memory and gated recursive units) can be used as the recursive model RM.

[0097] In a corresponding application of the recursive model RM, the model can take the values ​​of one or more measurable quantities from the corresponding states as input. For this recursive model, it is also possible to use values ​​from multiple states in a sliding window manner. For example, the figure shows a recursive model RM that takes the values ​​of measurable quantities of two objects at the current states xi,t713 and xj,t714, and the immediately preceding states xi,t-1711 and xj,t-1712.

[0098] Interestingly, by using a recursive model (RM), encoder models supporting both Markov and non-Markov systems can be obtained. As an example, an encoder model according to one embodiment can use the pairwise contribution function (PCF) shown in the figure in combination with: summation as an aggregation function, and another trainable function (e.g., a multilayer perceptron) as a node update function, for example: .

[0099] Here, the cascade for the RNN is along the feature axis.

[0100] Figure 7b Detailed, but non-limiting, examples of determining pairwise contributions used in encoder models are shown.

[0101] Similar to Figure 7a The figure illustrates the pairwise contribution function PCF721, which is used to determine the pairwise contribution pi,j731 of a given object to other objects based on the observed data Xi710. The observed data can represent a time series of states, where a state includes the values ​​of one or more measurable quantities of multiple interacting objects. (About...) Figure 6 The observational data were further discussed. According to Figure 5 In this architecture, the pairwise contribution function (PCF) can be used as a pairwise contribution function in the encoder model.

[0102] In this example, the pairwise contribution pi,j731 for a pair of two objects can be obtained as follows: the pairwise contribution pi,j,t737 at each time point is determined independently, and the pairwise contributions pi,j,t at each time point are combined to obtain the pairwise contribution pi,j, which is output by the pairwise contribution function PCF.

[0103] Specifically, the figure illustrates the time-point contribution function TCF726. It is typically a trainable function. The time-point contribution function TCF takes as input the values ​​of measurable quantities of two objects at a given time point, and optionally at a fixed number of immediately preceding time points. In this figure, the time-point contribution function TCF takes as input the current value and immediately preceding values ​​xi,t-1, xi,t of the measurable quantity of the first object, and the current value and immediately preceding values ​​xj,t-1, xj,t of the measurable quantity of the second object. For example, the time-point contribution function TCF could be a multilayer perceptron: .

[0104] To combine the pairwise contributions pi,j,t at each time point into the overall pairwise contribution pi,j, a suitable permutation covariance operator can be used. For example, this operator could involve summing the contributions pi,j,t at each time point, optionally followed by applying a trainable function (such as a multilayer perceptron), for example: .

[0105] By combining the contributions at each independently determined time point, the pairwise contribution function (PCF) may be particularly well-suited for modeling Markov systems, in which case it offers a particularly good trade-off between accuracy and computational cost.

[0106] For example, according to one embodiment, the encoder model can combine the pairwise contribution function (PCF) with summation as an aggregation function and other trainable functions (e.g., a multilayer perceptron) as node update functions, for example: .

[0107] Interestingly, the pairwise contribution function (PCF) described in the figure allows for robustness of the encoder model with respect to the permutation of observed time steps, while avoiding the vanishing gradient problem.

[0108] Now we turn to the training of the encoder and / or decoder models as described in this paper. Generally, to train the encoder and / or decoder, training data can be used to represent multiple observed scenes. A scene can include multiple observed states. The number of objects may vary between scenes, and the number of states may also vary between scenes. For example, the total number of available scenes may be at most or at least 500, at most or at least 10,000, or at most or at least 50,000. Similarly, the total number of states across all scenes may be at most or at least 50,000, at most or at least 250,000, at most or at least 1,000,000, or at most or at least 5,000,000.

[0109] Generally, the decoder model can be trained based on a set of training scenarios in the following way (e.g., regarding...). Figure 5 The described decoder model involves: repeatedly selecting a scene; predicting one or more states of the scene based on other states of the scene; deriving a training signal based on comparing the state of the scene with its prediction; and adapting trainable parameters based on the training signal.

[0110] For example, the encoder and decoder models can be trained together end-to-end in a conventional manner. Optimization can be performed, for example, through gradient descent of the objective function, etc. Here,f It is by v A parameterized decoder model, and g It is by w A parameterized encoder. In this example, the decoder model... f Configured to output the predicted difference to the immediately preceding state, the objective function compares the predicted difference with the actual difference. A comparison can be made. However, the decoder model can also alternatively output the predicted state.

[0111] However, it is of interest that in the various embodiments described below, the training of the encoder model and the decoder model is decoupled in the sense that the decoder model is trained without simultaneously training the encoder model. The encoder model can then be trained based on the trained decoder model. It is even possible to use the decoder model instead of the encoder model to determine the values ​​of the time-invariant latent features, as discussed elsewhere.

[0112] Figure 8a A detailed but non-limiting example of how to train a decoder model is shown. As illustrated in the figure, the example follows a general structure as follows: select a scene X810; determine a prediction x'811 for the state of the scene; derive a training signal TS820 based on comparing the state X with the prediction x'; and adapt the parameters V830 of the decoder model (e.g., a graphical model and / or local functions) based on the training signal TS.

[0113] Interestingly, in this example, the values ​​H890 of one or more time-invariant latent features of the objects in the scene are also trainable parameters jointly learned with the trainable parameters V of the decoder model. For example, in some or all iterations, both the decoder parameters V and the latent features H can be updated based on the same training signal TS. Alternatively, in some iterations, the decoder parameters V can be updated, while in other interleaved iterations, the latent features H can be updated. However, the parameters of the encoder model are not updated simultaneously; in fact, in this example, the encoder model is typically not used at all when training the decoder.

[0114] Typically, training is performed using iterative optimization methods, in which an objective function is optimized. For example, the objective function might quantify the prediction error, such as as L2 loss. Optimization can be stochastic, such as stochastic gradient descent. For instance, the Adam optimizer, as disclosed in Kingma and Ba's "Adam: A Method for Stochastic Optimization" (available at https: / / arxiv.org / abs / 1412.6980 and incorporated herein by reference), can be used. Such optimization methods can be heuristic and / or reach local optima, as is known. Training can be performed on a scene-by-scene basis or in batches (e.g., at most or at least 64 scenes, or at most or at least 256 scenes).

[0115] Interestingly, the training dataset can be partially annotated, meaning it includes values ​​of time-invariant latent features for at least some time-invariant features of at least some objects in at least some scenarios. These values ​​are then fixed, rather than being trained as trainable parameters. For a given latent feature, the training dataset can include any combination of fully annotated, partially annotated, and unannotated scenarios. In a fully annotated scenario, the latent feature values ​​are available for all objects. In a partially annotated scenario, the latent feature values ​​are available for some, but not all, objects. In an unannotated scenario, the latent feature values ​​are not available. The availability of latent feature values ​​may vary depending on the latent feature. However, for at least some latent features, at least some objects in some scenarios are not available, but are trained as trainable parameters. In general, any combination of available annotations is supported, and any combination of available annotations helps ensure that the time-invariant latent features are trained to derive some kind of information. For example, across the entire training data, the values ​​of a given time-invariant latent feature may be available for at most 20%, at most 10%, or even at most 1% of all objects in all scenarios.

[0116] Nevertheless, accurate semi-supervised learning of this time-invariant hidden feature can still be achieved.

[0117] However, unsupervised learning is also possible, or, for example, semi-supervised learning of some latent features can be combined with supervised learning of other latent features.

[0118] If discrete latent features are used, they can be trained as continuous parameters for efficiency reasons. For example, in the case of a single discrete latent feature using one-hot encoding, the discrete feature can be represented as... h ∈ H ,in During training, integrity constraints can be applied. h ∈H Relaxation In this context, it is advantageous if the training signal rewards trainable continuous parameters that approximate a discrete set of possible values. For example, a loss term can be used to penalize deviations from the set of possible values. This is particularly advantageous if the number of unlabeled latent features or labeled values ​​is relatively small. As observed by the inventors, in this case, trainable latent features will typically be clustered and need to be regularized to approximate the set of possible values.

[0119] Specifically, in order to train the parameters V and time-invariant latent features H of the decoder model, the following optimizations can be performed: Here, f v ( x t , h ) is the value of the measurable quantity predicted by the decoder model, and Δ x t Is it the actual value, or equivalent? f v ( x t , h ) is the difference between a measurable quantity and a value from a previous state, which is predicted by the decoder model, and Δ x t This is the actual difference. For example, in which... S Each of the training scenarios has the same number T In the specific case of a time step, optimization can be performed to minimize the loss, which represents the average norm of the prediction error across the time step and the training samples one step ahead (e.g., l 2 Norm): in It is a prediction made by the decoder, and These are the corresponding observed values.

[0120] The inventors have discovered that several measures, either individually or in combination, can be taken to significantly improve the robustness of this optimization.

[0121] In some embodiments, to improve robustness, the values ​​of the time-invariant latent feature vectors can remain constant over a number of initial iterations (e.g., for up to 5 or 10 iterations). For example, when using... K When encoding one-hot values ​​for a number of possible discrete values, a constant one-hot encoding can be used initially. h =(1 / K ...1 / K ).

[0122] In some embodiments, a regularization term may be included in the optimization that penalizes the deviation of discrete time-invariant latent features from the set of possible values. Besides allowing for better training of discrete features, this term has been found to improve robustness. It has been found that adding this term to the objective function, rather than initially excluding it (e.g., after a certain number of iterations, after it has been found that the latent features have been separated (e.g., clustered) to a sufficient degree, is particularly effective.

[0123] In some embodiments, to improve robustness, after a certain number of iterations in which time-invariant latent features are learned as trainable parameters, the latent features can be fixed for the remainder of the optimization, while the decoder continues to be trained. In particular, discrete latent features can be fixed to their learned relaxed discretization.

[0124] Interestingly, instead of fixing the latent features based on their values ​​as learned parameters, the inventors recognized that it is also possible to determine the fixed values ​​using a decoder DEC trained so far. Therefore, optimization can be performed to determine the values ​​of the time-invariant latent features that lead the decoder DEC to make the most accurate predictions. For example, it is possible to target a specific... To perform optimization ,in It is the decoder trained so far. f Iterative applications of DEC, the decoder predicts future... k There are several steps, and among them... This represents the current value of the decoder parameters. For example, the number of steps can be chosen to be at most or at least 5, at most or at least 10, or at most or at least 20. For example, optimization can be performed using an algorithm based on first-order gradients, or for discrete values, optimization can be performed using grid or beam-search or similar methods. The latter may be faster, depending on the number of objects and the number of possible values ​​for the discrete features, and may enable the use of functions... For decoder models that perform poorly.

[0125] As the inventors recognize, given a trained decoder model DEC, the aforementioned optimization for determining the values ​​of time-invariant latent features can be considered a type of encoder. For example, when applying a decoder, such as regarding... Figure 2 or Figure 6As discussed, time-invariant latent features can be determined by performing optimization based on the decoder model, for example, rather than by applying a separate encoder model. Therefore, it is possible and useful to train the decoder model (DEC) without subsequently training the encoder model. However, for efficiency reasons, a trained encoder model is generally preferred.

[0126] Figure 8b A detailed, but non-limiting, example of how to train the encoder model Enc801 is shown. For example, it can be done according to... Figure 8a This technique is applied after the decoder model has been trained.

[0127] In this example, the encoder model is trained in a supervised manner based on the training scene X810 and the corresponding known time-invariant latent feature values ​​H890 of the objects in that scene. The time-invariant latent features H used can correspond to values ​​learned as trainable parameters when training the decoder model. In the case of discrete latent features H, they are preferably discretized by selecting the closest value from the set of possible discrete values.

[0128] Based on the training scenario X, the encoder model Enc can be applied to determine the values ​​H' of the time-invariant latent features. These values ​​can then be compared with the known values ​​H to derive the training signal TS821. Based on this training signal, the parameters W831 of the encoder model can be adapted. Typically, the training of the encoder model can be performed by optimizing a loss function, such as the L2 norm of the difference between the determined values ​​H' and the known values ​​H, or cross-entropy.

[0129] Although the encoder model is trained in a supervised manner in this example, it should be noted that the overall training of the encoder and decoder models together is not a fully supervised learning task. This is because the overall training does not use the full set of ground truth labels for the latent feature values, but only, and optionally, a subset thereof. The encoder Enc can then be used to analyze new scenes by determining the time-invariant latent feature values ​​of the new scene.

[0130] Figure 8c This provides a detailed, but non-restrictive, example of how to train an encoder model. This example is based on... Figure 8b Example. Similar to this figure, the parameters W831 of the encoder model Enc801 can be trained in a supervised manner based on the following operations: the encoder model Enc is applied to scene X810 to obtain the values ​​H'891 of one or more time-invariant latent features; and these values ​​are compared with the known value H890' to derive the training signal TS821. Based on this training signal, the parameters W can be adapted.

[0131] This diagram illustrates the following situation: the known value H is not determined based on the values ​​of the latent feature vectors learned during the training of the decoder model. Instead, in this example, the known value H is determined during the optimization of Opt840. The optimized Opt selects values ​​H such that they yield the best prediction x'811 through the decoder model. Regarding... Figure 8a Detailed examples of this point were discussed.

[0132] Typically, the trainable components of the encoder and decoder models described in this paper (e.g., graphical models, pairwise contribution functions, node update functions, and local functions) can be implemented using various types of trainable functions. For example, trainable functions can be implemented using multilayer perceptrons or another type of neural network (e.g., convolutional neural networks, etc.). For example, the number of layers in the neural network used can be at least 2, at least 5, or at least 10, and the number of nodes and / or weights can be at least 1000 or at least 10000. In the case of neural networks, the trained parameters of the decoder and / or encoder networks can include the weights of the nodes in the neural network. Other types of trainable functions, such as Gaussian processes, may also be used. Typically, the number of trainable parameters in the encoder and / or decoder networks can be at least 1000 or at least 100000. From a training efficiency perspective, it is beneficial to use trainable functions that follow gradient-based optimization (e.g., are continuous and / or differentiable in their parameter set).

[0133] Several specific examples of encoder and / or decoder models as described herein are now presented. In these examples, the encoder and decoder models are applied to training data of interacting particles according to representations of three physical models. In the first example, the behavior of particles connected by a spring is modeled. In the second example, charged particles are modeled whose interactions are defined by Coulomb's law. In the third example, the behavior of a coupled oscillator is modeled according to the Kuramoto model. The design choices described below have been found to work well for this particular observational data, but may also be applicable to other types of observational data.

[0134] In these three examples, the encoder and decoder models are configured using an object with a discrete-time invariant latent feature, which has… K = 2 possible values. The measurable quantities are the object's position and velocity (in this example, in 2D). Below, x Used to represent particle position, and v Used to represent particle velocity. Therefore, the value of a measurable quantity of an object in a certain state can be represented as a four-dimensional vector ( x 1, x 2, v 1,v 2).

[0135] For the decoder model, use Figure 6 The architecture has the capability for Figure 6 Local functions and Figure 5 A multilayer perceptron (MLP) with pairwise contribution functions and node update functions. For the encoder model, based on... Figure 5 The graphical model is used in conjunction with an MLP as a node update function. The encoder model for charged particles and springs is based on... Figure 7b Markov encoder. Figure 7b The time-point contribution function is also implemented as an MLP function. The MLP in this example has two hidden layers, each with 32 neurons. A continuously differentiable exponential linear unit is used as the activation function for the MLP. For the Kuramoto example, a function based on... Figure 7a The recursive encoder. The recursive model is a single-layer Long Short-Term Memory (LSTM) module with a hidden size of 32. In both the encoder and decoder, pairwise contributions (i.e., ...) p i,j , i ≠ j ∈[ A ]) is a 32-dimensional vector.

[0136] The encoder and decoder models are trained using a two-step approach, with the Adam optimizer used in both steps at a learning rate of 0.001.

[0137] The training data used for the spring and Kuramoto examples represents scenarios where objects with equal hidden features interact. In the Coulomb example, particles with equal hidden features repel each other, and particles with different hidden features attract each other. The encoder and decoder models were trained using four different datasets. In the first dataset, all scenes have 2 particles. In the second dataset, all scenes have 5 particles. In the third dataset, half of the scenes have 4 particles, and half have 6 particles. The fourth dataset includes scenes with 3, 5, 7, and 9 particles, each representing a quarter of the dataset. Each of these models was applied to test data with 2, 3, 4, 5, 6, 7, 8, and 9 particles.

[0138] Interestingly, in the Spring and Coulomb examples, it was found that it is possible to train accurate encoder and decoder models in an unsupervised manner using training data consisting of 512 scenes. Even better results were obtained using training data consisting of 12,288 scenes. In a semi-supervised context, training was found to be effective with training data consisting of 640 scenes, where the ratio of labeled to unlabeled scenes was 1 to 9. For the Kuramoto example, the number of scenes was twice as many, but fewer states were used per scene. The trained model generalized beyond the number of objects used in the training data; for example, models trained with 4 or 6 particles were found to work well on test data with 3 particles.

[0139] Now, details are given regarding the training data used in these three examples. This training data was generated based on a synthesis of the physical model.

[0140] In the spring example, it is shown that A There are 36 particles, each associated with a discrete, random label. According to Hooke's Law, particles of the same type exert a force on each other, while particles of different types do not interact. In other words, particles... j For particles i The applied force F i,j Depend on This is determined by [the law of motion]. Furthermore, the motion of each particle is governed by Newton's laws.

[0141] In the example of charged particles, it is shown that A There are 10 particles, each with a random charge. q i The value is either -1 or 1. These particles interact according to a smoothed version of Coulomb's law and can therefore attract or repel each other. In fact, the particles... j For particles i The applied force F i,j Calculate using the following formula: in It refers to the relative positions of the particles, and ε >0 is the smoothing constant. Regarding the spring model, Newton's laws are used to determine the motion of the particles. The smoothing constant is set to... ε =0.01, to avoid the model having to effectively learn discontinuous functions. This can be difficult for the various practical choices of trainable functions used in encoder and decoder models.

[0142] In this example, bounding boxes are used when generating training data to prevent particles from drifting too far apart. The interaction between the bounding box and each particle is modeled by a force that is zero inside the box and increases linearly with distance beyond the box. For example, with x coordinate w The vertical boundary has x coordinate p Applying force to any particle - c ·max(0, p - w Specifically, it can be used c =10.

[0143] In the Kuramoto example, random intrinsic frequencies are utilized. ω i ∈[1,10) and random discrete type h i to indicate A One oscillator. Starting from a random initial phase. Starting at (the position on the unit circle), the dynamics of this system are governed by the following differential equation: This means that only oscillators of the same type are coupled.

[0144] All physical models are simulated using Euler integration with a step size of 0.01. Training examples are obtained by subsampling the resulting data every 10 time steps. This means that, in practice, the trajectories in the training data for this example have a step size of 0.1. For the spring and Coulomb examples, further subsampling can be performed, where subsampling is done only once every 10 time points. In this case, the final number of time steps in the scene can be set to 500. It should be noted that when using the one-step prediction error with a step size equal to the original step size as the training objective, the second subsampling step does not change the effective step size of the data. For the Kuramoto model, the data is kept at its original sampling frequency (10 in this example) and relatively short trajectories of 50 time steps are used. This prevents the phase locking effect known from this physical system. It should be noted that, generally, the model does not need the number of time steps to be constant across different scenes of the training data.

[0145] Figure 9 A block diagram is shown of a computer implementation method 900 for performing time series prediction on multiple interacting objects for controlling and / or monitoring a computer-controlled system. The corresponding objects may be part of the computer-controlled system or its environment. Method 900 may correspond to... Figure 2 The operation of system 200. However, this is not a limitation, because method 900 can also be performed using another system, apparatus or device.

[0146] Method 900 may include: in an operation entitled “accessing the decoder”, access 910 represents model data of a trained decoder model.

[0147] Method 900 may include, in an operation entitled “Obtaining Observation Results”, obtaining 920 observation data points representing a time series of one or more observation states of the plurality of interactive objects. The states may include values ​​of one or more measurable quantities of the plurality of interactive objects.

[0148] Method 900 may include: in an operation entitled “Determining Time-Invariant Latent Features”, determining the values ​​of one or more time-invariant latent features of the plurality of objects based on one or more observation states.

[0149] Method 900 may include, in an operation entitled "Predicting Next State", predicting 940 at least one next state of the plurality of interacting objects. To perform prediction 940, method 900 may include, in an operation entitled "Applying a Graph Model", applying 942 a trained graph model of a decoder model to obtain a first predicted contribution for the object. The first predicted contribution may be based on pairwise contributions of the object to a plurality of other objects. The pairwise contributions of the object to the other objects may be determined based on the values ​​of time-invariant latent features of the object and the other objects, and previous values ​​of one or more measurable quantities. To perform prediction 940, method 900 may further include, in an operation entitled "Applying a Function", applying 944 a trained function of a decoder model to obtain a second predicted contribution for the object. The second predicted contribution may be based on previous values ​​of one or more measurable quantities of the object itself, and optionally on the values ​​of time-invariant latent features of the object itself. To perform prediction 940, method 900 may further include, in an operation entitled "Combining Contributions", combining 946 the first and second predicted contributions to obtain predicted values ​​of one or more measurable quantities of the object.

[0150] Method 900 may include, in an operation entitled “generating output data”, generating 950 output data based on at least one predicted next state for use in controlling and / or monitoring a computer-controlled system.

[0151] Figure 10A block diagram of a computer implementation method 1000 for training a decoder model is shown. This decoder model is used for time series prediction of multiple interactive objects, which are related to a computer-controlled system in the sense that the corresponding objects can be part of the computer-controlled system or its environment. Method 1000 may correspond to... Figure 1 The operation of system 100. However, this is not a limitation, because method 1000 can also be performed using another system, apparatus or device.

[0152] Method 1000 may include: in an operation entitled “Accessing Training Data”, accessing training data representing multiple observed scenes, 1010.

[0153] Method 1000 may include training a decoder model by performing one or more of a plurality of operations 1030-1060.

[0154] To train the decoder model, method 1000 may include, in an operation entitled "Selecting a Scene," selecting 1030 of the observed scenes from the plurality of observed scenes. The scene may represent a time series of multiple observed states of multiple interacting objects. The state may include the value of one or more measurable quantities of the plurality of interacting objects. To train the decoder model, method 1000 may include, in an operation entitled "Obtaining Time-Invariant Latent Features," obtaining 1040 the values ​​of one or more time-invariant latent features of the plurality of objects.

[0155] To train the decoder model, method 1000 may include: in an operation entitled "Predicting State", predicting 1050 the state of the observed scene based on a previous state of the observed scene. To predict this state, method 1000 may include: in an operation entitled "Applying a Graph Model", applying 1052 a graph model of the decoder model to obtain a first predicted contribution for the object. The first predicted contribution may be based on pairwise contributions of the object with multiple other objects. The pairwise contributions of the object with other objects may be determined based on the values ​​of time-invariant latent features of the object and other objects, as well as previous values ​​of one or more measurable quantities. To predict this state, method 1000 may include: in an operation entitled "Applying a Function", applying 1054 a function of the decoder model to obtain a second predicted contribution for the object. The second predicted contribution may be based on previous values ​​of one or more measurable quantities of the object itself, and optionally on the values ​​of time-invariant latent features of the object itself. To predict this state, method 100 may include: in an operation entitled "Combining Contributions", combining 1056 the first and second predicted contributions to obtain predicted values ​​of one or more measurable quantities of the object.

[0156] To train the decoder model, method 1000 may include, in an operation entitled "Deriving Training Signal," deriving a 1060 training signal based on comparing the observed state of the scene with its prediction. To train the decoder model, method 1000 may include, in an operation entitled "Adapting Parameters," adapting the parameters of the 1070 graphical model and / or the function based on the training signal.

[0157] Method 1000 may optionally further include training an encoder model as described herein.

[0158] Figure 11 A block diagram is shown of a computer implementation method 1100 for analyzing observations of multiple interacting objects for controlling and / or monitoring a computer-controlled system. The corresponding objects may be part of the computer-controlled system or its environment. Method 1100 may correspond to... Figure 2 The operation of system 200. However, this is not a limitation, because method 1100 can also be performed using another system, apparatus or device.

[0159] Method 1100 may include, in an operation entitled “Accessing Model Data,” accessing 1110, which represents model data of a trained encoder model. The encoder model may be trained in conjunction with the decoder model described herein.

[0160] Method 1100 may include, in an operation entitled “Obtaining Observation Data”, obtaining 1120 observation data, which represents a time series of one or more observation states of the plurality of interactive objects. The states may include values ​​of one or more measurable quantities of the plurality of interactive objects.

[0161] Method 1100 may include, in an operation entitled “Determining Time-Invariant Latent Features”, using an encoder model 1130 to determine the values ​​of one or more time-invariant latent features of the plurality of objects based on the one or more states. This determination may include applying a trained graphical model of the encoder model based on pairwise contributions of the object to multiple other objects. The pairwise contributions of the object to the other objects may be determined based on the values ​​of one or more measurable quantities of the object and the other objects included in one or more observation states.

[0162] Method 1100 may include, in an operation entitled “Generating Output Data”, generating 1140 output data based on the values ​​of one or more time-invariant hidden features for use in controlling and / or monitoring a computer-controlled system.

[0163] What will be understood is, generally speaking Figure 9 Method 900 Figure 10 Method 1000 and Figure 11The operations of method 1100 can be performed in any suitable order, such as sequentially, simultaneously, or a combination thereof, where applicable, the order being subject to a specific order necessary, for example, by input / output relationships. Some or all of these methods can also be combined; for example, method 900 and / or method 1100 can be performed using an encoder or decoder model previously trained according to method 1000.

[0164] This method(s) can be implemented on a computer as a computer-implemented method, dedicated hardware, or a combination of both. Similarly, Figure 12 As illustrated, instructions for a computer (e.g., executable code) can be stored on a computer-readable medium 1200, for example, in the form of a machine-readable physical tag sequence 1210 and / or as a sequence of elements having different electrical (e.g., magnetic) or optical properties or values. The executable code can be stored in a temporary or non-temporary manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 12 An optical disc 1200 is shown. Alternatively, the computer-readable medium 1200 may include temporary or non-temporary data 1210, which is model data representing a decoder model trained according to the methods described herein and / or an encoder model trained according to the methods described herein.

[0165] Examples, embodiments, or optional features, whether or not indicated as non-limiting, should not be construed as limiting the claimed invention.

[0166] It should be noted that the above embodiments are illustrative rather than limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments without departing from the scope of the appended claims. Any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The use of the verb "comprising" and its variations does not exclude the presence of elements or stages other than those recited in the claims. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Expressions such as "at least one" when preceding a list or group of elements indicate the selection of all elements or any subset of elements from that list or group. For example, "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In device claims enumerating several components, several of these components can be implemented by the same item of hardware. The mere fact that certain measures are described in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously.

Claims

1. A computer-implemented prediction method for performing time-series prediction on multiple interacting objects for controlling and / or monitoring a computer-controlled system, wherein the respective objects are part of the computer-controlled system or its environment, wherein the computer-controlled system is an autonomous or semi-autonomous vehicle, the method comprising: - Access model data representing the trained decoder model; - While the vehicle is operating in the environment: - Using a camera integrated in the vehicle to acquire observation data, the observation data representing a time series of one or more observation states of the plurality of interactive objects, each observation state including the value of one or more measurable quantities of the plurality of interactive objects, wherein the plurality of interactive objects include multiple traffic participants in the environment of the vehicle; - Determine the value of one or more time-invariant hidden features of the plurality of interactive objects for each object based on the one or more observation states; - Predict the next state of the multiple interacting objects using the following steps: - A trained graph model of the decoder model is applied to obtain a first predicted contribution for an object, which is based on the pairwise contributions of the object with a plurality of other objects. The pairwise contributions of the object with each other object are determined based on the values ​​of the time-invariant latent features of the object and the other objects and the previous values ​​of one or more measurable quantities. - Apply a trained function of the decoder model to obtain a second predictive contribution for the object, the second predictive contribution being based on previous values ​​of one or more measurable quantities of only the object and not any other object among the plurality of interacting objects, and based on the values ​​of one or more time-invariant latent features of at most the object and not any other object among the plurality of interacting objects, wherein the trained function does not take into account any interaction with any other object when obtaining the second predictive contribution; - The first and second prediction contributions are combined to obtain predictions of one or more measurable quantities of the object; - Generate output data based on the predicted next state for use in controlling and / or monitoring the computer-controlled system; as well as - Use the output data to control the vehicle, the control including controlling the steering and / or braking of the vehicle; - where each time-invariant latent feature is a feature that the decoder model assumes to be invariant within each time window for predicting its next state.

2. The prediction method of claim 1, wherein the values ​​of one or more time-invariant latent features of the object are determined based on a trained encoder model, the determination comprising: A trained graphical model of the encoder model is applied based on the pairwise contributions of the object and multiple other objects, the pairwise contributions of the object and each other object being determined based on the values ​​of one or more measurable quantities of the object and the other objects included in the one or more observation states.

3. The prediction method according to claim 2, wherein the time-invariant latent features are discrete features with a finite set of possible values, and determining the time-invariant latent features includes: The output of the trained encoder model is discretized into the finite set.

4. The prediction method of claim 1, wherein one or more measurable quantities include one or more of the object's position, velocity, and acceleration.

5. A computer-implemented training method for training a decoder model for time-series prediction of multiple interacting objects associated with a computer-controlled system, said computer-controlled system being an autonomous or semi-autonomous vehicle, wherein the corresponding objects are part of said computer-controlled system or its environment, said method comprising: - Access training data representing multiple observed scenes; - The decoder model is trained by performing the following steps over multiple iterations: - Select the observed scene from the plurality of observed scenes, where the observed scene represents a time series of multiple observed states of multiple interactive objects, each observed state including the value of one or more measurable quantities of the plurality of interactive objects, wherein the plurality of interactive objects include multiple traffic participants in the environment of the vehicle; - Obtain the values ​​of one or more time-invariant implicit features based on the plurality of interactive objects for each object; - Predict the state of the observed scene based on its previous states using the following steps: - Apply the graph model of the decoder model to obtain a first predicted contribution for the object, which is based on the pairwise contributions of the object with a plurality of other objects. The pairwise contributions of the object with each other object are determined based on the values ​​of the time-invariant latent features of the object and the other objects and the previous values ​​of one or more measurable quantities. - Apply a trained function of the decoder model to obtain a second predictive contribution for the object, the second predictive contribution being based on previous values ​​of one or more measurable quantities of only the object and not any other object among the plurality of interacting objects, and based on the values ​​of one or more time-invariant latent features of at most the object and not any other object among the plurality of interacting objects, wherein the trained function does not take into account any interaction with any other object when obtaining the second predictive contribution; - The first and second prediction contributions are combined to obtain predictions of one or more measurable quantities of the object; - A training signal is derived by comparing the observed state of the scene with the predicted state, and the parameters of the graphical model and / or the function are adapted based on the training signal; - where each time-invariant latent feature is a feature that the decoder model assumes to be invariant within each time window for predicting the state it is in; - The training includes training the decoder model to output data on the next state of the observed scene, predicted by the decoder using observation data from cameras integrated in the vehicle, for computer control of the autonomous or semi-autonomous vehicle.

6. The method of claim 5, further comprising training an encoder model, the encoder model being configured to determine values ​​of one or more time-invariant latent features for each object based on one or more observation states of the plurality of interacting objects, the determination comprising: A trained graphical model of the encoder model is applied based on the pairwise contributions of the object and multiple other objects, the pairwise contributions of the object and each other object being determined based on the values ​​of one or more measurable quantities of the object and the other objects included in the one or more observation states.

7. The method of claim 6, wherein training the decoder model comprises: As trainable parameters, the values ​​of one or more time-invariant latent features of the objects in the scene observed by each object are jointly learned; The method further includes using learned values ​​of one or more time-invariant latent features to perform supervised training of the encoder model.

8. The method of claim 7, wherein the training data comprises the values ​​of one or more time-invariant latent features of a subset of objects and / or time-invariant latent features and / or observed scenes.

9. A computer-implemented analytical method for analyzing observations of multiple interacting objects for controlling and / or monitoring a computer-controlled system, wherein the respective objects are part of the computer-controlled system or its environment, the method comprising: - Access model data representing a trained encoder model, which is trained in conjunction with a decoder model according to the method of claim 6; - Obtain observation data, which represents a time series of one or more observation states of the plurality of interactive objects, each observation state including the value of one or more measurable quantities of the plurality of interactive objects; - Using an encoder model to determine the values ​​of one or more time-invariant latent features of the plurality of interacting objects based on the one or more states, the determination comprising: applying a trained graph model of the encoder model based on the pairwise contributions of the object to a plurality of other objects, the pairwise contributions of the object to each other object being determined based on the values ​​of one or more measurable quantities of the object and the other objects included in the one or more observation states; - Generate output data based on the values ​​of one or more time-invariant hidden features for use in controlling and / or monitoring the computer-controlled system.

10. The method of claim 9, wherein generating the output data comprises: Further machine learning models can be applied based on the values ​​of the time-invariant latent features.

11. A predictive system for time-series prediction of multiple interacting objects for controlling and / or monitoring a computer-controlled system, wherein the respective objects are part of the computer-controlled system or its environment, wherein the computer-controlled system is an autonomous or semi-autonomous vehicle, the system comprising: - Data interface for accessing model data representing the trained decoder model; - The processor subsystem is configured to: - While the vehicle is operating in the environment: - Using a camera integrated in the vehicle to acquire observation data, the observation data representing a time series of one or more observation states of the plurality of interactive objects, each observation state including the value of one or more measurable quantities of the plurality of interactive objects, wherein the plurality of interactive objects include multiple traffic participants in the environment of the vehicle; - Determine the value of one or more time-invariant hidden features of the plurality of interactive objects for each object based on the one or more observation states; - Predict the next state of the multiple interacting objects using the following steps: - A trained graph model of the decoder model is applied to obtain a first predicted contribution for an object, which is based on the pairwise contributions of the object with a plurality of other objects. The pairwise contributions of the object with each other object are determined based on the values ​​of the time-invariant latent features of the object and the other objects and the previous values ​​of one or more measurable quantities. - Apply a trained function of the decoder model to obtain a second predictive contribution for the object, the second predictive contribution being based on previous values ​​of one or more measurable quantities of only the object and not any other object among the plurality of interacting objects, and based on the values ​​of one or more time-invariant latent features of at most the object and not any other object among the plurality of interacting objects, wherein the trained function does not take into account any interaction with any other object when obtaining the second predictive contribution; - The first and second prediction contributions are combined to obtain predictions of one or more measurable quantities of the object; - Generate output data based on the predicted next state for use in controlling and / or monitoring the computer-controlled system; as well as - Use the output data to control the vehicle, the control including controlling the steering and / or braking of the vehicle; - where each time-invariant latent feature is a feature that the decoder model assumes to be invariant within each time window for predicting its next state.

12. A training system for training a decoder model for time-series prediction of multiple interacting objects associated with a computer-controlled system, said computer-controlled system being an autonomous or semi-autonomous vehicle, wherein the corresponding objects are part of said computer-controlled system or its environment, said system comprising: - Data interface for accessing training data representing a scene with multiple observations; - The processor subsystem is configured to train the decoder model by performing the following steps over multiple iterations: - Select the observed scene from the plurality of observed scenes, where the observed scene represents a time series of multiple observed states of multiple interactive objects, each observed state including the value of one or more measurable quantities of the plurality of interactive objects, wherein the plurality of interactive objects include multiple traffic participants in the environment of the vehicle; - Obtain the values ​​of one or more time-invariant implicit features based on the plurality of interactive objects for each object; - Predict the state of the observed scene based on its previous states using the following steps: - Apply the graph model of the decoder model to obtain a first predicted contribution for the object, which is based on the pairwise contributions of the object with a plurality of other objects. The pairwise contributions of the object with each other object are determined based on the values ​​of the time-invariant latent features of the object and the other objects and the previous values ​​of one or more measurable quantities. - Apply a trained function of the decoder model to obtain a second predictive contribution for the object, the second predictive contribution being based on previous values ​​of one or more measurable quantities of only the object and not any other object among the plurality of interacting objects, and based on the values ​​of one or more time-invariant latent features of at most the object and not any other object among the plurality of interacting objects, wherein the trained function does not take into account any interaction with any other object when obtaining the second predictive contribution; - The first and second prediction contributions are combined to obtain predictions of one or more measurable quantities of the object; - A training signal is derived by comparing the observed state of the scene with the predicted state, and the parameters of the graphical model and / or the function are adapted based on the training signal; - where each time-invariant latent feature is a feature that the decoder model assumes to be invariant within each time window for predicting the state it is in; - The training includes training the decoder model to output data on the next state of the observed scene, predicted by the decoder using observation data from cameras integrated in the vehicle, for computer control of the autonomous or semi-autonomous vehicle.

13. An analytical system for analyzing observations of multiple interacting objects for controlling and / or monitoring a computer-controlled system, wherein the respective objects are part of the computer-controlled system or its environment, the system comprising: - A data interface for accessing model data representing a trained encoder model, which is trained in conjunction with a decoder model according to the method of claim 6; - The processor subsystem is configured to: - Obtain observation data, which represents a time series of one or more observation states of the plurality of interactive objects, each observation state including the value of one or more measurable quantities of the plurality of interactive objects; - Using an encoder model to determine the values ​​of one or more time-invariant latent features of the plurality of interacting objects based on the one or more states, the determination comprising: applying a trained graph model of the encoder model based on the pairwise contributions of the object to a plurality of other objects, the pairwise contributions of the object to each other object being determined based on the values ​​of one or more measurable quantities of the object and the other objects included in the one or more observation states; - Generate output data based on the values ​​of one or more time-invariant hidden features for use in controlling and / or monitoring the computer-controlled system.

14. A computer-readable medium comprising temporary or non-temporary data representing one or more of the following: - When executed by a processor system, it causes the processor system to execute the instructions of the computer-implemented method according to claim 1 and / or claim 5 and / or claim 9; - Represents model data of a decoder model trained according to the computer implementation method of claim 5 and / or an encoder model trained according to the computer implementation method of claim 6.

Images