IMPROVEMENTS IN A SWITCHING RECURRING KALMAN NETWORK

Abstract

A computer-implemented method for controlling a device, comprising the following: Receiving data x t from a sensor; Encoding the data x t via parameters of an encoder (102) to make a latent observation w t , for the data x t and an uncertainty vector σ w t for the latent observation w t to obtain; Processing the latent observation w t using a recurrent neural network to control a switching variable s t , which the weights α t determined to obtain a locally linear Kalman filter; Processing the latent observation w t and the uncertainty vector σ w t using the locally linear Kalman filter to obtain updated mean values ​​µ z t a latent representation z t of the Kalman filter and covariances Σ z t the latent representation z tto obtain the Kalman filter; Decoding the latent representation z t of the Kalman filter, to obtain an average value, µ x t , and a covariance Σ x t a reconstruction of data x t to obtain; Output of the data reconstruction x t at a time t; and Controlling the device based on the reconstruction of data x t .

Description

Technical field

[0001] This disclosure refers generally to a system and a procedure for estimating unknown variables from measurements observed over time in a machine learning system. background

[0002] A linear quadratic estimation (LQE), commonly known as a Kalman filter, is an algorithm that produces estimates of unknown variables based on a series of measurements observed over time. These measurements may contain noise and other inaccuracies, so the estimates of the unknown variables can be more accurate than those based on a single measurement, as the estimate incorporates a joint probability distribution over the variables for each time frame.

[0003] DE 11 2022 004 057 T5 discloses a computer-implemented method that utilizes a continuous discrete recurrent Kalman network, the method comprising: receiving, at an encoder, an input from one or more sensors, wherein the input comprises one or more time-series data that associate data at one or more time points; outputting, at a Kalman filter, a latent observation and an uncertainty estimate in response to the input at the encoder; determining a latent prior state and a latent posterior state using the Kalman filter; and outputting, via a decoder, a filtered observation that utilizes at least the latent posterior state.

[0004] Becker, P., Pandya, H., Gebhardt, G., Zhao, Ch., Taylor, CJ., and Neumann, G., Proceedings of the 36th International Conference on Machine Learning, PMLR 97:544-552, 2019, propose a new deep approach to Kalman filtering that is learned directly on an end-to-end basis using backpropagation. The approach employs a high-dimensional factored latent state representation for which Kalman updates are simplified to scalar operations to avoid hard-to-propagate, computationally intensive, and potentially unstable matrix inversions. Locally linear dynamical models are used to propagate the latent state to the next time step.

[0005] Shaj, V., Becker, P., Büchler, D., Pandya, H., van Duijkeren, N., Taylor, CJ, Hanheide, M., and Neumann, G., Proceedings of the 2020 Conference on Robot Learning, PMLR 155:765-781, 2021, use a probabilistic recurrent neural network architecture called Recurrent Kalman Networks to model learning. Two architectures are presented: one for forward model learning and one for inverse model learning.

[0006] DE 11 2021 000 422 T5 concerns the prediction of future trajectories in environments with multiple actuators for autonomous machine applications. Systems and methods are disclosed that utilize previously tracked actuator positions and map information to predict future actuator positions using a deep neural network, such as a recurrent neural network. In addition to previous positions and map information, waiting conditions, free-space boundary information, and / or other environmental information can be used by the system to compute accurate and reliable predictions of future actuator positions. Summary

[0007] The invention is defined by the independent claims. The dependent claims define advantageous embodiments. A method for controlling a device includes receiving data from a first sensor, encoding the data via parameters of an encoder to enable latent observation (w t ) for the data and an uncertainty vector (σw) t ) to obtain the latent observation, processing the latent observation with a recurrent neural network to create a switching variable (s t ), the weights (α t ) of a locally linear Kalman filter, to obtain, the processing of the latent observation and the uncertainty vector with the locally linear Kalman filter to obtain an updated mean of the latent representation (µ) Zt ) and a covariance of the latent representation (Σ Zt ) of the Kalman filter, decoding the latent representation to obtain a mean value (µ)Xt ) and a covariance (Σ Xt ) to obtain a reconstruction of the data, and to output the reconstruction at a time t.

[0008] A device control system includes a control unit. The control unit can be configured to receive data from a first sensor, encode the data via parameters of an encoder, and perform latent monitoring (w). t ) for the data and an uncertainty vector (σw) t ) to obtain the latent observation, processing the latent observation with a recurrent neural network to create a switching variable (s t ), the weights (α t ) of a locally linear Kalman filter, to obtain, processing the latent observation and the uncertainty vector with the locally linear Kalman filter to obtain an updated mean (µ) Zt ) of the latent representation and a covariance (Σ Zt) of the latent representation (Zt) of the Kalman filter, decoding the latent representation to obtain a mean (µ) Xt ) and a covariance (Σ Xt ) to obtain a reconstruction of the data, and output the reconstruction at a time t.

[0009] A system for processing time series data includes an encoder, a Kalman update block, a locally linear Kalman filter, an inference network, a closed recurrent unit, and a decoder. The encoder can be configured to process an observation (x t ) to receive and a uncertainty vector (σw t ) and a latent observation (w t ) to output. The Kalman update block can be configured to receive the uncertainty vector and the latent observation and to output a mean of the latent representation (µ). Zt ) and a covariance (Σ Xt) of the latent representation. The locally linear Kalman filter can be configured with weights (α). t ), to receive the earlier mean and covariance of the latent representation and to output the later mean and covariance of the latent representation. The inference network can be configured to receive the latent observation and a deterministic recurrent cell (h). t ) to receive and a switching variable (s t ) and output weights for the locally linear Kalman filter. The closed recurrent unit can be configured to receive the switching variable and output the deterministic recurrent cell. The decoder can be configured to receive the latent representation and output a mean (µ). Xt ) of the latent observation and a covariance (Σ Xt ) to output the latent observation. Brief description of the drawings Fig. Figure 1 is a flowchart of a switching recurrent Kalman network (SRKN). Fig. Figure 2 is a data flow diagram of the switching recurrent Kalman network of Fig. 1. Fig. Figure 3 is a graphical representation of the motion trajectories generated by the switching recurrent Kalman net. Fig. Figure 4 is a graphical representation of image sequences generated by the switching recurrent Kalman net based on the first two time steps. Fig. Figure 5 is a block diagram of an electronic computing system configured to run the switching recurrent Kalman net. Fig. Figures 6-9 are graphical representations of motion trajectories generated by the switching recurrent Kalman net based on initial observation. Fig.Figure 10 is a schematic diagram of a control system configured to control a vehicle. Fig. Figure 11 is a schematic diagram of a control system configured to control a manufacturing machine. Fig. Figure 12 is a schematic diagram of a control system configured to control a power tool. Fig. Figure 13 is a schematic diagram of a control system configured to control an automated personal assistant. Fig. Figure 14 is a schematic diagram of a control system configured to control a monitoring system. Fig. Figure 15 is a schematic diagram of a control system configured to control a medical imaging system. Detailed description

[0010] As required, specific embodiments of the present invention are disclosed here; however, the disclosed embodiments are, of course, merely exemplary of the invention, which may be implemented in various and alternative forms. The illustrations are not necessarily to scale; some features may be exaggerated or reduced in size to show details of specific components. Therefore, the specific structural and functional details disclosed here are not to be interpreted as limiting, but merely as a representative basis for teaching a person skilled in the art how to use the disclosed invention in various ways.

[0011] The term "essentially" may be used here to describe disclosed or claimed embodiments. The term "essentially" may modify a value or relative property disclosed or claimed in the present disclosure. In such cases, "essentially" may mean that the value or relative property it modifies is within ± 0%, 0.1%, 0.5%, 1%, 2%, 3%, 4%, 5%, or 10% of the value or relative property.

[0012] The term sensor refers to a device that detects or measures and records, indicates, or otherwise reacts to a physical property. The term sensor encompasses an optical, light, image, or photon sensor (e.g., a charge-coupled device (CCD), a CMOS active pixel sensor (APS), an infrared (IR) sensor, a CMOS sensor), an acoustic, sound, or vibration sensor (e.g., microphone, geophone, hydrophone), an automotive sensor (e.g., wheel speed, parking, radar, oxygen, blind spot, torque, lidar), a chemical sensor (e.g., an ion-sensitive field-effect transistor (ISFET), oxygen, carbon dioxide, chemiresistor, holographic sensor), a sensor for electric current, electric potential, magnet, or radio frequency (e.g., Hall effect, magnetometer, magnetoresistance, Faraday cup, galvanometer), an environmental, weather, humidity, or humidity sensor (e.g.,weather radar, actinometer), a flow or fluid velocity sensor (e.g., air mass flow sensor, anemometer), a sensor for ionizing radiation or subatomic particles (e.g., ionization chamber, Geiger counter, neutron detector), a navigation sensor (e.g., a global positioning system sensor (GPS sensor), a magnetohydrodynamic sensor (MHD sensor), a position, angle, displacement, distance, velocity, or acceleration sensor (e.g., LIDAR, accelerometer, ultra-wideband radar, piezoelectric sensor), a force, density, or level sensor (e.g., strain gauge, nuclear density meter), a heat, temperature, or thermal sensor (e.g.,Infrared thermometer, pyrometer, thermocouple, thermistor, microwave radiometer) or any other device, module, machine or subsystem whose purpose is to detect or measure a physical property and to record, indicate or otherwise respond to it.

[0013] In particular, a sensor can measure properties of a time-series signal and can incorporate spatial or spatiotemporal aspects, such as a location in space. The signal can contain electromechanical, acoustic, light, electromagnetic, RF, or other time-series data. The technology disclosed in this application can be applied to time-series imaging with other sensors, such as an antenna for wireless electromagnetic waves, a microphone for sound, etc.

[0014] The term "image" refers to a representation or artifact that depicts the perception of a physical property (e.g., audible sound, visible light, infrared light, ultrasound, underwater acoustics), such as a photograph or other two-dimensional image that resembles an object (e.g., a physical object, a scene, or a plot of land) and thus provides a representation of it. An image can be multidimensional in that it may contain components of time, space, intensity, concentration, or other properties. For example, an image can be a time-series image. This technology can also be extended to image 3D sound sources or objects.

[0015] Predicting driving behavior or other sensor measurements is an essential component of autonomous and semi-autonomous driving systems. Real-world multivariate time-series data are often difficult to model because the underlying dynamics are nonlinear and the observations are subject to noise. Additionally, driving data can often be multimodally distributed, meaning that distinguishable predictions exist that are probable, but averaging can negatively impact model performance. This disclosure presents a switching recurrent Kalman network (SRKN) for efficient inference and prediction of nonlinear and multimodal time-series data. The architecture of this network switches between multiple Kalman filters that model different aspects of the dynamics in a factorized latent state.The architecture and the resulting scalable and interpretable deep state-space model were tested on toy datasets and real-world taxi driving data in Porto, Portugal. In all cases, the model captured the multimodal nature of the dynamics in the data.

[0016] One implementation, such as predicting a vehicle's trajectory, which is a key competency for future autonomous driving, is considered. Predicting the future trajectory refers to estimating the future state of certain agents based on their past measurements.

[0017] This capability is crucial for autonomous vehicles to plan safe future navigation and avoid potential risks. Predicting is a difficult task due to the inherent ambiguity and uncertainty in forecasting future trajectories. For example, at any given time in a traffic scenario, there are multiple destinations a driver could pursue, and there are several plausible routes to reach each destination. These destinations are often not observable from the outside, making the future both non-deterministic and multimodal. Averaging the dynamics is typically inadequate and, in many cases, physically difficult. Consider a scenario where an obstacle is located in the lane a car is traveling in. To avoid the obstacle, the vehicle can move to the left or right lane.Averaging these two possible maneuvers results in the car crashing directly into the obstacle. Autonomous vehicles must be aware of these multiple possibilities to navigate safely through urban areas.

[0018] A common approach to modeling time-series data is state-space models. State-space models rely on latent states whose transition dynamics determine the system's behavior and are linked to measurements through a noisy observation process. A Kalman filter is frequently used in a state-space model. It is a preferred solution for deriving linear Gaussian systems. However, real-world time-series data are often nonlinear, and the data generation process is typically unknown. Unfortunately, prior inference in nonlinear, non-Gaussian systems is generally uncontrollable. Several efforts have been made in the deep learning community to overcome the problem of nonlinearity and system identification. Two common approaches are either using approximations to make nonlinear systems controllable or introducing stochastics into recurrent neural networks.

[0019] The recurrent Kalman network (RKN) is an efficient probabilistic recurrent neural network architecture that uses Kalman updates to infer the system state. In general, RKNs follow the first approach, mapping observations onto a latent feature space where Kalman updating is feasible. To overcome nonlinearity, RKNs maintain a bank of linear basis systems that can be interpolated over time.

[0020] This revelation presents an alternative approach to predicting future trajectories that takes multimodality and uncertainty into account. Specifically, it employs a recurrent Kalman network with a variational inference technique to introduce a deep learning model capable of modeling multimodal dynamics. This model offers the interpretability of a state-space model while scaling well for real-time inference and prediction tasks.

[0021] This disclosure will present data to demonstrate the proposed models on a real-world task: modeling the trajectory of a taxi. Traffic forecasting, due to its nonlinear temporal and spatial dependencies, is a stimulating problem for autonomous driving. Understanding this traffic behavior is crucial for urban traffic monitoring and electronic traffic distribution.

[0022] Machine learning often employs a Bayesian framework to quantify the degree of uncertainty surrounding an event. Bayesian modeling uses probabilities to systematically assess model uncertainties. A well-known example of combining Bayesian modeling and deep learning is variational autocoders (VAEs). These are unsupervised deep learning models that attempt to find a compressed representation of observations in a latent space. VAEs have seen widespread use and have been extended to incorporate temporal dependencies.

[0023] Time series data are often described by state-space models. State-space models assume the existence of an underlying system that governs the process of observation generation. This system evolves over time, leading to temporal dependencies in the observations. In state-space models, both the observations and the underlying system states are modeled using probability distributions. The concept of the state-space model dates back to the 1960s, when the Kalman filter was introduced for linear and Gaussian systems. Despite its elegant computation and simplicity, the Kalman filter is limited to linear and Gaussian state-space models. A number of papers in the tax theory community propose solving multimodality and nonlinearity problems by maintaining a bank of K linear systems and interpolating between them.However, these methods often require knowledge of system parameters and are not designed for working with high-dimensional data.

[0024] Deep state-space models, while manageable, are often not sufficiently detailed to capture multimodality. Nonlinear deep state-space models have emerged as an alternative, but they lose their manageability and must rely on approximation techniques. Although all these deep state-space models are successful in modeling complex real-world time-series data, they are not explicitly designed to capture multimodality.

[0025] A recurrent Kalman network is a probabilistic recurrent neural network architecture for sequential data that uses Kalman updates to learn a representation of a latent state of the system. It achieves competitive results on various state estimation tasks while providing reasonable uncertainty estimation and efficiency. In this work, we propose combining a recurrent Kalman network with a switching Kalman filter to address the multimodal dynamics of time-series data.

[0026] Fig. Figure 1 is a flowchart of a switching recurrent Kalman network (SRKN) 100. The SRKN contains an encoder 102, an update block 104, a Kalman filter 106, an inference network 108, a closed recurrent unit cell 110 and a decoder 112.

[0027] Fig.1 is also referred to as the architecture of the switching recurrent Kalman network. The encoder forms the observations (x t ) on a space of latent features (w t ) off. The encoder also produces an uncertainty vector for the mapped latent observations. A closed recurrent unit cell is present, which contains information about the switching variable (s tThe latent observation is stored over time. The latent observation is combined with the GRU cell to approximate the later distribution for the transition variable. A single sample from this later distribution is passed to a softmax layer to produce the weight coefficients for the transition basis matrices. The later distribution of the latent state from the previous time step is combined with the weighted basis matrices to form the predicted distribution for the current latent state. The resulting prediction is then filtered using the latent observation and its uncertainty vector in the Kalman update step. Finally, a single sample from the later distribution is fed into the decoder to parameterize the approximated distribution for the current observation.

[0028] A switching recurrent Kalman network (SRKN) is an extension of a recurrent Kalman network that incorporates multimodality. The architecture of the model is described in Fig. Figure 1 visualizes this. The SRKN establishes a latent observation space and a latent state space. Observations, such as images, are mapped onto a latent observation space in which linear dynamics are feasible. The transformation into this latent feature space is provided by the SRKN encoder and can be learned end-to-end. In this latent space, precise subsequent inference can be performed using the Kalman filter.

[0029] The generative model in latent space. The latent state space Z = R 25 is linked to the latent observation by a simple linear emission function, which is represented in Equation 1: wt=Hzt; H=[Im 0m×m] where m is the dimensionality of the latent observation, I mdenotes the identity matrix and 0 m×m This represents an m × m matrix filled with zeros. This emission model effectively splits the latent state vector into two parts. The first (upper) part contains information inherent in the observation, and the second (lower) part, the memory, is information derived over time, such as velocities. Depending on the input dimension (images or reality), an uncertainty vector is also output by the decoder.

[0030] Fig. Figure 2 is a data flow diagram of the switching recurrent Kalman network of Fig. 1. This figure represents a generative model 200 and an inference model 250 of the switching recurrent Kalman net.

[0031] In the generative model, the switching variable s t on their distribution up to the current time step and the previous latent state z tconditioned. The deterministic recurrent cell h t stores information about s t over time. s t determines the weights of the basis matrices. The linear model at time step t is a weighted sum of the basis systems. The current latent state is related to the previous latent state by a linear model, where the switching variable is given. The observation x t is decoupled from the latent state. In the inference model, the dependence on s t from z t-1 rejected. In addition, the real observations are reduced to a latent representation w t depicted. w t is used for the inference of s t and z t used. This has the advantage that the inference of z t is available in closed form with the Kalman filter.

[0032] The generative model in the observation space. The decoder f decparameterizes the distribution of the reconstructed observation using a single sample of the latent state, as shown in Equation 2: p(xt|zt,st)=N(μxt,∑xt) with [μxt,∑xt]=fdec(zt);zt∼p(zt|st,zt−1).

[0033] The transition model. The SRKN assumes that the system dynamics evolve locally linearly over time. In this way, the system state can be derived online using a Kalman filter. To obtain locally linear transition dynamics, the SRKN maintains a bank of transition basis matrices A. (k) , and the transition matrix at each time step is a weighted sum of these base matrices. The prediction distribution for the latent state at time step t is represented by equation 3: At=∑k=1Kαt(k)A(k);αt=(αt(1),...,αt(K))=softmax(st);∑k=1Kαt(k)=1; αt(k)≥0 p(zt|st,zt−1)=N(μzt−,∑zt−) with μzt−=Atμzt−1+;∑zt−=At∑zt−1+AtT+I.σtrans,

[0034] Here, µ denotes z-t and Σ z-t the previous mean and previous covariance of z t , while μzt−1+ and ∑zt−1+ the mean and covariance of the later of the preceding latent state z t-1 represent. Furthermore, there is αt(k) The weight assigned to the k-th linear basis matrix is ​​called the switch variable. Its value is non-negative, and all weights sum to one. The idea for multiple transition basis matrices is very similar to the switching Kalman filter. The weights assigned to the transition basis matrices are represented by the switch variable s. t given.

[0035] This switching variable depends on its distribution in the preceding time steps and on the latent state of the preceding time step. Therefore, a closed recurrent unit g is used to store information about the switching variable over time. A neural network f trans is used to combine information from the latent state and toggling variables, as shown in Equation 4: p(st|s <t,zt−1)=N(μst,∑st) mit[μst,∑st]=ftrans(ht,zt−1); ht=g(ht−1,st−1) αt=softmax(st); st∼N(μst,∑st).

[0036] The weight coefficients for the base matrices are obtained by taking a sample of s t through a softmax layer. In summary, the generative model is factored as shown in Equation 5: p(x1:T,z1:T,s1:T)=∏t=1Tp(xt|st,zt)p(zt|st,zt−1)p(st|s <t,zt−1).

[0037] The inference model: This revelation presents the following factorization of the inference model, as shown in Equation 6: q(s1:T,z1:T|x1:T)=∏t=1Tq(zt|st,zt−1,xt)q(st|s <t,xt) q(st|s<t,xt)=N(μst,∑st) mit [μst,∑st]=finf(s<t,xt)p(zt|st,zt−1,xt)=N(μzt+,∑zt+) mit[μzt+,∑zt+]=Kalman_Aktualisierung(μzt−,∑zt−).

[0038] The inference for z t is given by a factored Kalman update introduced by the RKN. Here, the condition of s t on z t-1 rejected, see Fig. 2. Empirical experiments show that removing this condition from the inference model solves the problem of mode averaging when training the model.

[0039] The inference of the switching variables is performed using amortized variational inference, in which the inference networks and the generative networks are trained together. These networks are responsible for parameterizing the probability distributions of the switching variables and the observations. The inference of the latent system state also follows the elegant computational structure of the RKN, in which the filtering process can be simplified to scalar operations.

[0040] The lower bound of evidence: This model belongs to the class of approaches via variation. The variational inference technique formulates a steerable lower bound for the complex distribution of interest, thus transforming the approximation of a difficult-to-solve later distribution into an optimization problem. This is achieved by finding an approximate later distribution that minimizes its KL divergence from the real later distribution. Minimizing the KL divergence is equivalent to maximizing the following lower bound of evidence (ELBO), as shown in Equation 7. LELBO=∑t=1TEq(zt|st,zt−1,fw(xt))q(st|s <t,fw(xt))[log p(xt|st,zt)]−Eq(st|s<t,zi−1,fw(xt−1))[Eq(zt−1|st−1,zi−2,fw(xt−1))[KL(q(zt|st,zt−1,fw(xt)‖p(zt|st,zt−1))]]−Eq(s1|fw(x1))[...Eq(st|s<t,zt−1,fw(xt))[Eq(zt−1|st−1,zt−2,fw(xt−1))[KL(q(st|s<t,zt−1,fw(xt))‖p(st|s<t,zt−1))]]].

[0041] Here, f refers to w the function that represents the real observation x ton the latent observation w t This revelation introduces a scaling factor for each component of the ELBO. These scaling factors are motivated by the β-VAE and determine the trade-off between the reconstruction term and the regularization term. Depending on the given problems, tuning these scaling factors can be useful for overall training performance. In addition, we add a prediction loss term to guide the model's training process. This prediction loss term is the weighted sum of K observation probabilities. Each probability p (k) (x t |s t ,z t-1 ) refers to the probability of observation when the transition of the latent state z occurs. t the linear basis system A (k)The following follows. Intuitively, the prediction loss term corresponds to the logarithmic probability of a mixed model with K components. The prediction loss term forces the model to assign a higher weight to the basis systems that are more likely to produce the subsequent observation. The resulting objective function is shown in Equation 8. LObjective=Lβ_ELBO+βpredLPred, with Lpred=∑t=1Tlog∑k=1Kαt(k)p(k)(xt|st,zt−1)with p(k)(xt|st,zt−1)=Ep(k)(zt|st,zt−1)[p(xt|st,zt)p(k)(zt|st,zt−1)] p(k)(zt|st,zt−1)=N(zt;A(k)zt−1,A(k)∑zt−1(A(k))T).

[0042] L β_ELBO refers to the ELBO, where the reconstruction loss term, the KL divergence for z t and the KL divergence for s t a scaling factor β rec , β z or β s exhibit.

[0043] The SRKN was evaluated using several datasets. First, a simulated 2D time-series dataset with four dynamic modes and a synthetic image dataset of vehicle movements following an underlying structure were considered. Then, the SRKN was applied to a real taxi dataset. The results were then compared with various time-series data modeling methods incorporating RKN, VRNN-GMM, VDM, and DMM-IAF.

[0044] Evaluation metrics: Four metrics were selected for the quantitative evaluation of the predictions. They included i) one-stage prediction loss logp(x). t |x <t ), ii) multi-stage prediction loss logp(x t:t+τ |x <t ), iii) logarithmic reconstruction probability logp(x t |x ≤t) and iv) Wasserstein distance. A real observation is modeled with a multivariate Gaussian distribution with diagonal covariance. The negative logarithmic Gaussian reconstruction probability for a sequence in this case is shown by Equation 11. L(x1:T)=1T∑t=1T−log N(xt|μxt+,σxt+)

[0045] The negative high-dimensional data are modeled using a Bernoulli distribution. The logarithmic reconstruction probability is calculated as shown in Equation 12. L(x1:T)=−1T∑t=1T∑d=0Dxt(d)log(μxt(d)+)+(1−μxt(d)+) log(1−μxt(d)+)

[0046] The single-stage prediction loss term demonstrates the prediction performance of the model for the next time step, with the given observations up to the current time step, as shown by Equation 13. Lone_step(x1:T)=∑t=1T−1−log p(xt+1|x1:t).

[0047] To calculate the multi-stage prediction loss, n = 100 predictions are generated for the remainder of the sequence, using the given observations up to time step τ, as shown in Equation 14. Lmulti_steps(x1:T)=1n∑i=1n∑t=τT−1−log p(i)(xt+1|x1:τ)

[0048] The Wasserstein distance takes into account both the diversity and the accuracy of the prediction. To approximate the Wasserstein distance, n samples are selected from a test set exhibiting similar initial motion trajectories. The model is expected to produce sample predictions that agree with all ground-truth continuations in the test set, given the initial motion trajectories.

[0049] Fig.Figure 3 is a graphical representation of the motion trajectories generated by the switching recurrent Kalman net. Figure 6 represents the motion trajectories generated by the SRKN. Figures 1-2 and 1-3 show different transition modes that the model assigns to each possible continuation of the motion trajectory. Each element 302, 304, 306, 308, 310 corresponds to a dynamic transition mode. Each time step is coded in grayscale (302, 304, 306, 308, 310) with the mode to which the model assigns the highest weight.

[0050] Fig.Figure 4 is a graphical representation of image sequence 400, generated by the switching recurrent Kalman network based on the first two time steps (t-1 and t). Two image sequences (402 and 404) were generated by the switching recurrent Kalman network based on the first two time steps (t-1 and t). Each grayscale level corresponds to a dynamic transition mode. Each image is encoded in grayscale using the mode to which the model assigned the highest weight. The two rectangles are not present in the dataset but are used for visualization purposes only. Here, the model can determine the two potential trajectories that the car can follow as it approaches the intersection.

[0051] Fig.Figure 5 is a block diagram of an electronic computing system configured to execute the switching recurrent Kalman network. This electronic computing system may also include a telecommunications system, a machine architecture, and a machine-readable medium. Fig. Figure 5 is a block diagram of an electronic computing system suitable for implementing the systems or executing the methods disclosed herein. The machine in Fig.Figure 5 is shown as a standalone device suitable for implementing the concepts described above. For the server aspects described above, multiple such machines, operating in a data center, as part of a cloud architecture, etc., can be used. Not all of the functions and devices shown are used in server aspects. For example, while a system, device, etc., used by a user to interact with a server and / or cloud architectures may have a screen, touchscreen input, etc., servers often do not have screens, touchscreens, cameras, and so on, and typically interact with users through connected systems that have appropriate input and output capabilities.Therefore, the architecture described below should be understood as encompassing several types of devices and machines, and various aspects may or may not exist in any given device or machine, depending on its form factor and purpose (for example, servers rarely incorporate cameras, while body-worn devices rarely include magnetic disks). The example explanation is in . Fig. However, paragraph 5 is suitable to enable experts to determine how the embodiments described above can be implemented with a suitable combination of hardware and software with a suitable modification of the embodiments shown on the specific device, machine, etc. used.

[0052] Although only a single machine is depicted, the term "machine" should be understood to include a collection of machines that, individually or collectively, execute a group of (or several groups of) instructions to carry out one or more of the methodologies discussed here.

[0053] The example for Machine 500 includes at least one processor 502 (e.g., a control unit, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), an extended processing unit (APU), or combinations thereof), one or more memories such as main memory 504, static memory 506, or other memory types, which communicate with each other via a connection 508. The connection 508 may be a bus or another type of communication channel. Machine 500 may further include optional aspects, such as a graphics display unit 510, which comprises any type of display device. Machine 500 may also include other optional aspects, such as an alphanumeric input device 512 (e.g., a keyboard, a touchscreen, and so on), a user interface (UI) navigation device 514 (e.g., a navigation bar ...a mouse, a trackball, a touch-sensitive device, and so on), a storage unit 516 (e.g., a disk drive or other storage device(s)), a signal-generating device 518 (e.g., a loudspeaker), sensor(s) 521 (e.g., a global position sensor, accelerometer, microphone(s), camera(s), and so on), an output control unit 528 (e.g., a wired or wireless connection for connecting and / or communicating with one or more other devices, such as a universal serial bus (USB), near field communication (NFC), infrared (IR), serial / parallel bus, etc.), and a network interface device 520 (e.g., wired and / or wireless) for connecting to and / or communicating over one or more networks 526.

[0054] The various memories (e.g., 502, 506, and / or the memory of the processor(s) 502) and / or the memory unit 516 can store one or more instruction sets and data structures (e.g., software) that embody or are used by one or more of the methodologies or functions described herein. When executed by the processor(s) 502, these instructions effect various operations for implementing the disclosed embodiments. Toy experiments

[0055] Synthetic 2D dataset. It starts with a simple two-dimensional dataset to verify the proposed model's ability to capture multimodality. Each sequence consists of five time steps. The data sequences have a constant value in the first three steps. At time step 4, each dimension of the data point can switch between two possible modes, so the data has a total of four modes. Fig. Figure 3 is a visualization of the results. The model successfully captures the switchover point in the fourth time step.

[0056] Synthetic image dataset of vehicle motion trajectories. Next, the SRKN is evaluated using a simple synthetic image dataset of vehicle motion trajectories. The observations here are image sequences of 24 × 24 pixels. The black square represents a vehicle whose motion trajectory follows an underlying pattern containing two adjacent rectangles. Each image represents the vehicle's position at a specific time. The vehicle does not move in the opposite direction at any time step. The qualitative results are presented in Fig.Figure 4 demonstrates this. Each image is encoded with the dominant mode predicted by the model. The black square appears blurry in later time steps, presumably due to transition noise incorporated into the model. It is noteworthy that, although trained only on sequences of length 6, the models can also make good predictions for longer sequences. In other words, the models can learn and generalize the underlying dynamics of the data. Thus, one possible application of the SRKN is to model real-world image data of motion trajectories in autonomous driving. It should be noted that the two rectangles are not part of the dataset but are included for evaluation purposes only.

[0057] The quantitative results for the toy experiments are given in Table 4. This model achieved competitive results compared to the VDM on the four-mode dataset, while it achieved the best one-step and multi-step prediction performance on the pendulum image dataset. Four-mode data set Vehicle movement path data set Step 1 multi-step w-abst. LL Step 1 multi-step w-abst. LL VDM -4,83 2,11 0,10 -4,90 7,04 7,45 6,44 6,23 RKN -3,91 3,41 0,22 -4,83 4,33 5,33 7,11 2,63 VRNN -3,96 2,59 0,13 -5,06 5,14 8,14 6,21 4,93 DMM -2,94 4,00 0,72 -5,21 7,86 8,04 6,44 6,87 SRKN -4,12 2,37 0,10 -5,07 4,33 5,10 4,40 2,74

[0058] Table 1: Quantitative results for four datasets of four modes and vehicle motion paths. In the four-mode dataset, the SRKN and the VDM exhibit the smallest Wasserstein distance. This indicates their similar performance in predicting and capturing multimodality. Compared to the RKN, the SRKN achieves lower single-step and multi-step prediction loss. Of all the base models, only the VDM shows better single-step and multi-step prediction loss than the SRKN. In the vehicle motion path dataset, the SRKN outperforms all the base models in terms of prediction loss and Wasserstein distance. The reconstruction loss of the RKN is slightly better than that of the SRKN in this image dataset.

[0059] Fig.Figures 6-9 are graphical representations of motion trajectories generated by the switching recurrent Kalman network based on different initial observations. 50 motion trajectories were generated (thin line) for given initial observations (bold line). The model can generate motion trajectories that follow the general developmental structure of the underlying map.

[0060] Real-world taxi dataset: To validate the effectiveness of the proposed model, an experiment was conducted using a dataset from a single taxi in Porto. The original dataset consists of 1.7 million records from 442 taxis operating in Porto, Portugal. The preprocessing pipeline was reused for the evaluation. Only the movement paths within the city were selected, and only the first 30 time steps were extracted. The resulting dataset is divided into a training set of 86,386 records, a validation set of 200 records, and a test set of 10,000 records. Fig.Figure 6 demonstrates the qualitative forecasting results. The task is to predict the next 20 time steps based on the first 10. The model can capture the multimodal dynamics and provide predictions that follow the underlying developmental structure of the map. Compared to modern models for multimodality, such as VDM, SRKN cannot achieve such good forecast results. This could be because SRKN uses a linear state transition model, while the state transition in VDM is nonlinear and represented by a powerful deep neural network. Taxi data set Step 1 multi-step w-abst. LL Number of parameters VDM -3,68 2,88 0,59 -4,33 22056 RKN -2,9 4,2 2,07 -4,25 23118 VRNN -2,77 5,51 2,43 -4,09 22352 DMM -2,45 3.29 0,7 -4,35 22248 SRKN -2,35 3,16 0,75 -4,34 33742

[0061] Table 2: Quantitative results for taxi data sets. VDM outperforms all base models in terms of prediction loss and Wasserstein distance. Compared to RKN, SRKN shows a significantly lower Wasserstein distance and multistep prediction loss. This demonstrates an improvement of SRKN compared to RKN in long-term and multimodal prediction performance.

[0062] A switching recurrent Kalman network for multimodal modeling of time series data is presented above. The model consists of a recurrent neural network for the switching variable and a locally linear state transition model. It operates with a latent observation space in which a linear transition model is feasible. This necessitates the assumption of a state-space model and employs an explicit concept of the system state. The inference of the system state follows the efficient computational structure of the recurrent neural network, while the inference of the switching variable is performed using an amortized variational inference procedure. The model demonstrates the ability to capture multimodality using the real-world dataset of taxi trajectories in Porto. Furthermore, our model exhibits the interpretability of a state-space model with switching regimes and outperforms basic models on high-dimensional vehicle trajectory data.The ability of this model to incorporate uncertainty and multimodality into future predictions promises a wide range of applications in the field of autonomous driving, such as predicting the movement path of pedestrians and vehicles in the vicinity.

[0063] This technology can also be applied to other serial data, such as in the Fig. provided. The Fig.References 10-15 present exemplary embodiments, but the concepts of this disclosure can be applied to other embodiments. Some exemplary embodiments include: industrial applications, where the modalities may include video, weight, IR, 3D camera, and sound; power tool or appliance applications, where the modalities may include torque, pressure, temperature, distance, or sound; medical applications, where the modalities may include ultrasound, video, CAT scan, MRI, or sound; robotic applications, where the modalities may include video, ultrasound, LiDAR, IR, or sound; and security applications, where the modalities may include video, sound, IR, or LiDAR. The modalities may have different data sets; for example, a video data set may contain an image, a LiDAR data set may contain a point cloud, and a microphone data set may contain a time series.

[0064] The technology disclosed here can be used by working with time-series data obtained by receiving sensor signals, such as GPS signals from vehicles or emissions from power engines. Accurate predictive models of typical driving behavior, typical pollutant levels over time, or the dynamics of a power engine can help legislators and / or automotive engineers develop solutions for cleaner mobility. Other exemplary applications include:

[0065] Video classification: Existing methods are used to extract frame-based features from the video (e.g., object tracking). Based on these frame-based features, a predictive model is trained. For unseen videos, the video detection model (VDM) can predict plausible continuations of the features after viewing the initial frames (and extracting the features). These predictions can be used for video classification. These predicted features are fed into a classifier, with different effects depending on the use case (e.g., predicting traffic, predicting an impending accident (and if an accident is likely, dispatching emergency responders), predicting a violent / non-violent scene (and if violence is likely, shutting down the video)).

[0066] Autonomous Driving: External Model: Using sensor measurements (e.g., video, LiDAR, communication with other intelligent vehicles or smart city devices) to extract features about other road users and surrounding objects. Features could include 3D world coordinates, coordinates relative to the vehicle itself, surrounding objects, and other road users. Furthermore, this data can be used to train the Vehicle Dynamics Model (VDM) on these extracted features. A trained model can then be used in a vehicle: When new sensor measurements are recorded, features must be extracted, and these can then be predicted into the future by the VDM. These predictions can trigger various ECU behaviors (e.g., deceleration, emergency braking, etc.).

[0067] A driver model can use sensor measurements (e.g., video, steering, braking, communication with the driver's smartwatch) to extract features about the driver. These features include steering, acceleration, eye movements, and heart rate. The VDM can be trained on these extracted features. A trained model can then be used in a vehicle: as new sensor measurements are recorded, features must be extracted, and these can then be predicted into the future by the VDM. These predictions can trigger various ECU behaviors (e.g., deceleration, emergency braking, etc.).

[0068] A force machine model can use sensor measurements (e.g., from the ECU) to extract features about the force machine dynamics. These features include any of the ECU parameters and derived quantities. The force machine model (FDM) can be trained on such extracted features. A trained model can then be used in a vehicle, and when new sensor measurements are recorded, the features that need to be extracted can be predicted into the future by the vehicle dynamic model (VDM). These predictions can trigger various ECU behaviors (e.g., deceleration, emergency braking, etc.).

[0069] The battery state of health (SOH) or battery charge level (SOC) is used to track track features and features of driver behavior (e.g., speed and elevation of the track), and the VDM can be trained on such features.

[0070] The Internet of Things (IoT) (e.g., smart buildings, smart manufacturing) allows a system to collect and track sensor measurements and use these and derived values ​​as features. For some of these features, the system can have defined critical thresholds (e.g., minimum oxygen content, maximum temperature, etc.). When new measurements arrive, the system can use the Virtual Diagnostic Measure (VDM) to generate a forecast. If a critical threshold is likely to be exceeded within a specified timeframe, the system can take emergency action (e.g., stop the production line, open a valve to allow fresh oxygen in, open windows, lock safety doors).

[0071] Digital twins can be used as prototypes for new technical devices (e.g., power tools, household appliances, new power machine designs, etc.) and collect data from the device's internal sensors and / or external sensors (e.g., video, LiDAR) during normal use. These measurements and / or derived values ​​can be used as features to train the VDM (Virtual Device Model) on these features. Predicted behavior can be used to detect anomalies in the device's behavior (e.g., excessive energy consumption, premature shutdown, overheating, etc.). If undesirable behavior is expected during operation, the device can be automatically shut down, or its settings can be switched to safe mode.

[0072] Resource allocation, where a system measures the demand at different nodes of a network (e.g., a computer network, a telecommunications network, a wireless network). The system could then use this data, along with other measurements at the nodes (e.g., temperature, time of day) and / or derived variables, as features to record and train a VDM model. With new data, the system can then use the VDM to predict demand. For example, if it is predicted that the demand at a specific node will exceed a critical threshold, additional resources are allocated. Besides resource allocation, load forecasting is also needed for traffic management and routing algorithms. At each access point of a wireless network, resources such as spectrum and transmit power are highly limited and are allocated as needed. In short, based on the user's application type (e.g.,Based on the needs of an IoT or mobile user, the quality of service requirements (e.g., data rate, reliability, latency), the condition of the communication channel (signal-to-noise ratio), and other factors, the resource allocation unit assigns the appropriate transmission time slot, frequency, power, and transmission format. A good load forecasting algorithm is helpful for the timely allocation of resources, such as reserving frequencies if latency-critical traffic is anticipated. To serve the ever-increasing number of users with stricter quality of service requirements, load forecasting and resource allocation are becoming increasingly sophisticated in 5G and beyond.

[0073] Fig.Figure 10 is a schematic diagram of the control system 1102, configured to control a vehicle, which may be a partially autonomous vehicle or a partially autonomous robot. The vehicle includes a sensor 1104 and an actuator 1106. The sensor 1104 may include one or more wave-energy-based sensors (e.g., a charge-coupled CCD or video device), radar, LiDAR, a microphone array, ultrasound, infrared, thermal imaging, acoustic imaging, or other technologies (e.g., positioning sensors such as GPS). One or more of these specific sensors may be integrated into the vehicle. Alternatively or additionally to the specific sensors identified above, the control module 1102 may include a software module configured to determine a state of the actuator 1104 during execution.

[0074] In embodiments where the vehicle is at least partially autonomous, the actuator 1106 can be implemented in a braking system, a drive system, a power unit, a powertrain, or a steering system of the vehicle. Actuator control commands can be determined so that the actuator 1106 is controlled in such a way that the vehicle avoids collisions with detected objects. Detected objects can also be classified according to what the classifier considers most likely, e.g., pedestrians or trees. The actuator control commands can be determined depending on the classification. For example, the control system 1102 can classify an image (e.g., optical, acoustic, thermal) or other inputs from the sensor 1104 into one or more background classes and one or more object classes (e.g., pedestrians, bicycles, vehicles, trees, traffic signs, traffic lights, road debris, or construction containers / cones, etc.).The control system 1102 can segment an image into one or more background classes and one or more marking classes (e.g., lane markings, guardrails, road edges, vehicle lanes, etc.) and send control commands to the actuator 1106, implemented here in a steering system, to instruct the vehicle to avoid crossing markings and remain within its lane. In a scenario where an adversary attack might occur, the system described above can also be trained to better detect objects or to identify changes in lighting conditions or the angle of view of a sensor or camera on the vehicle.

[0075] In other embodiments, where the vehicle 1100 is an at least partially autonomous robot, the vehicle 1100 can be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot can be an at least partially autonomous lawnmower or a at least partially autonomous cleaning robot. In such embodiments, the actuator control command 1106 can be configured to control a drive unit, a steering unit, and / or a braking unit of the mobile robot so that the mobile robot can avoid collisions with identified objects.

[0076] In another embodiment, the vehicle 1100 is an at least partially autonomous robot in the form of a gardening robot. In such an embodiment, the vehicle 1100 can use an optical sensor, namely sensor 1104, to determine the condition of plants in an environment near the vehicle 1100. The actuator 1106 can be a nozzle configured to spray chemicals. Depending on an identified plant species and / or an identified condition of the plants, the actuator control command 1102 can be determined to cause the actuator 1106 to spray the plants with an appropriate quantity of suitable chemicals.

[0077] The vehicle 1100 can be a at least partially autonomous robot in the form of a household appliance. Non-restrictive examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 1100, the sensor 1104 can be an optical or acoustic sensor configured to detect the state of an object that is to be processed by the household appliance. For example, if the household appliance is a washing machine, the sensor 1104 can detect the state of the laundry inside the washing machine. The actuator control command can then be determined based on the detected state of the laundry.

[0078] In this embodiment, the control system 1102 would receive data or an image (optical or acoustic) from the sensor 1104. The control system 1102 can then process the information in Fig.The procedure described in section 1 is used to formulate a prediction of the image received by sensor 1104. Based on this prediction, signals can be sent to actuator 1106, for example, to brake or turn to avoid collisions with pedestrians or trees, to steer to stay between detected lane markings, or for any other actions performed by actuator 1106 as described above. Signals can also be sent to sensor 1104 based on this classification, for example, to focus or move a camera lens.

[0079] Fig.Figure 11 shows a schematic diagram of the control system 1202, which is configured to control the system 1200 (e.g., a manufacturing machine), such as a punching machine, a cutting machine, or a deep-hole drill, of the manufacturing system 102, such as part of a production line. The control system 1202 can be configured to control the actuator 14, which is configured to control the system 100 (e.g., the manufacturing machine).

[0080] The sensor 1204 of the system 1200 (e.g., the production machine) can be a wave energy sensor, such as an optical or acoustic sensor, or a sensor array configured to detect one or more properties of a manufactured product. The control system 1202 can be configured to determine the state of a manufactured product from one or more of the detected properties. The actuator 1206 can be configured to control the system 1202 (e.g., the production machine) for a subsequent manufacturing step of the manufactured product, depending on the determined state of the manufactured product 104. The actuator 1206 can be configured to perform the functions of Fig. 10 (e.g., the manufacturing machine) in subsequently manufactured products of the system (e.g., the manufacturing machine) depending on the specific state of the previous product.

[0081] In this embodiment, the control system 1202 would receive data or an image (e.g., optical or acoustic) and annotation information from the sensor 1204. The control system 1202 can then process this information. Fig. The procedure described in section 1 is used to formulate a prediction of the image received by sensor 1104. Based on this prediction, signals can be sent to actuator 1206. For example, if control system 1202 detects anomalies in a product, actuator 1206 can mark or remove anomalous or defective products from the line. In another example, if control system 1202 detects the presence of barcodes or other objects to be placed on the product, actuator 1106 can apply or remove these objects. Based on this classification, signals can also be sent to sensor 1204, for example, to focus or move a camera lens.

[0082] Fig. Figure 12 shows a schematic diagram of the control system 1302, which is configured to control the power tool 1300, such as a drill or a power screwdriver, which has at least a partially autonomous operating mode. The control system 1302 can be configured to control the actuator 1306, which is configured to control the power tool 1300.

[0083] The sensor 1304 of the power tool 1300 can be a wave energy sensor, such as an optical or acoustic sensor, configured to detect one or more properties of a work surface and / or a fastener driven into the work surface. The control system 1302 can be configured to determine a state of the work surface and / or the fastener relative to the work surface from one or more of the detected properties.

[0084] In this embodiment, the control system 1302 would receive an image (e.g., optical or acoustic) and annotation information from the sensor 1304. The control system 1302 can then process this information. Fig.The procedure described in section 1 is used to formulate a prediction of the image received by sensor 1304. Based on this prediction, signals can be sent to actuator 1306, for example, for the pressure or speed of the tool, or for any of the actions performed by 1306 as described in the preceding sections. Based on this classification, signals can also be sent to sensor 1304, for example, to focus or move a camera lens. In another example, the image can be a time-series image of signals from power tool 1300, such as pressure, torque, speed, temperature, current, etc., where the power tool is a rotary hammer, drill, (rotary or demolition) hammer, impact wrench, reciprocating saw, or oscillating multi-tool, and the power tool is either cordless or corded.

[0085] Fig.Figure 13 shows a schematic diagram of the control system 1402, which is configured to control an automated personal assistant 1401. The control system 1402 can be configured to control the actuator 1406, which is configured to control the automated personal assistant 1401. The automated personal assistant 1401 can be configured to control a household appliance, such as a washing machine, stove, oven, microwave, or dishwasher.

[0086] In this embodiment, the control system 1402 would receive an image (e.g., optical or acoustic) and annotation information from the sensor 1404. The control system 1402 can then process this information. Fig.The procedure described in section 1 is used to formulate a prediction of the image received by sensor 1404. Based on this prediction, signals can be sent to actuator 1406 to, for example, control moving parts of the automated personal assistant 1401, to interact with household appliances, or to control one of the actions performed by actuator 1406, as described in the preceding sections. Signals can also be sent to sensor 1404 based on this classification, for example, to focus or move a camera lens.

[0087] Fig.Figure 14 shows a schematic diagram of the control system 1502, which is configured to control a monitoring system 1500. The monitoring system 1500 can be configured to physically control access through door 252. The sensor 1504 can be configured to detect a scene relevant to the decision regarding granting access. The sensor 1504 can be an optical or acoustic sensor, or a sensor array, configured to generate and transmit image and / or video data. Such data can be used by the control system 1502 to detect a person's face.

[0088] The monitoring system 1500 can also be an observation system. In such an embodiment, the sensor 1504 can be a wave energy sensor, such as an optical sensor, an infrared sensor, or an acoustic sensor, configured to detect a scene to be observed, and the control system 1502 is configured to control the display device 1508. The control system 1502 is configured to determine a scene classification, e.g., whether the scene detected by the sensor 1504 is suspicious. A disturbance object can be used to detect specific types of objects, enabling the system to identify such objects under less than optimal conditions (e.g., at night, in fog, rain, with disturbing background noise, etc.). The control system 1502 is configured to send an actuator control command to the display device 1508 in response to the classification.The display device 1508 can be configured to adapt the displayed content in response to the actuator control command. For example, the display device 1508 can highlight an object that the control unit 1502 considers suspicious.

[0089] In this embodiment, the control system 1502 would receive an image (optical or acoustic) and annotation information from the sensor 1504. The control system 1502 can then process the information in Fig.The procedure described in section 1 is used to formulate a prediction of the image received by sensor 1504. Based on this prediction, signals can be sent to actuator 1506 to, for example, lock or unlock doors or other entrances, activate an alarm or other signal, or perform any of the actions carried out by actuator 1506 as described in the preceding sections. Signals can also be sent to sensor 1504 based on this classification, for example, to focus or move a camera lens.

[0090] Fig.Figure 15 shows a schematic diagram of the control system 1602, which is configured to control the imaging system 1600, for example, an MRI scanner, an X-ray imaging device, or an ultrasound device. The sensor 1604 can be, for example, an imaging sensor or an acoustic sensor array. The control system 1602 can be configured to determine a classification of all or part of the acquired image. The control system 1602 can be configured to determine or select an actuator control command in response to the classification obtained by the trained neural network. For example, the control system 1602 can interpret an area of ​​an (optical or acoustic) image as potentially anomalous. In this case, the actuator control command can be determined or selected to cause the display device 1606 to display the image and highlight the potentially anomalous area.

[0091] In this embodiment, the control system 1602 would receive an image and annotation information from the sensor 1604. The control system 1602 can then process this information. Fig. Use the procedure described in section 1 to formulate a prediction of the image received by sensor 1604. Based on this prediction, signals can be sent to actuator 1606, for example, to detect anomalous areas of the image or for any of the actions performed by 1606 as described in the preceding sections.

[0092] The program code implementing the algorithms and / or procedures described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium containing computer-readable program instructions that instruct a processor to execute aspects of one or more embodiments. Computer-readable storage media, which are non-transient in nature, can include volatile and non-volatile, as well as removable and non-removable, physical media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.Computer-readable storage media may also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, portable compact disc read-only storage (CD-ROM) or other optical storage media, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be read by a computer. Computer-readable program instructions can be downloaded from a computer-readable storage medium to a computer, another type of programmable data processing device or other device, or over a network to an external computer or external storage device.

[0093] Computer-readable program instructions stored on a computer-readable medium can be used to instruct a computer, other types of programmable data processing equipment, or other devices to operate in a specific manner, such that the instructions stored on the computer-readable medium produce a manufactured product containing instructions for implementing the functions, actions, and / or operations specified in the flowcharts or diagrams. In specific alternative embodiments, the functions, actions, and / or operations specified in the flowcharts and diagrams can be rearranged, processed serially, and / or processed simultaneously in accordance with one or more embodiments.Furthermore, each of the flowcharts and / or diagrams may contain more or fewer nodes or blocks than shown, in accordance with one or more embodiments.

[0094] Although the entire invention has been presented by describing various embodiments, and although these embodiments have been described in considerable detail, it is not the applicant's intention to limit or restrict the scope of protection of the appended claims to such details. Additional advantages and modifications are readily apparent to those skilled in the art. The invention in its broader sense is therefore not limited to the specific details, representative apparatus and methods, and illustrative examples shown and described. Accordingly, deviations from these details are permissible without departing from the spirit or scope of protection of the general concept of the invention.

Claims

[1] Computer-implemented method for controlling a device, comprising: Receiving data x t from a sensor; Encoding the data x t via parameters of an encoder (102) to make a latent observation w t , for the data x t and an uncertainty vector σ wt for the latent observation w t to obtain; Processing the latent observation w t using a recurrent neural network to control a switching variable s t , which the weights α t determined to obtain a locally linear Kalman filter; Processing the latent observation w t and the uncertainty vector σ wt using the locally linear Kalman filter to obtain updated mean values ​​µ zt a latent representation z t of the Kalman filter and covariances Σ zt the latent representation z tto obtain the Kalman filter; Decoding the latent representation z t of the Kalman filter, to obtain an average value, µ xt , and a covariance Σ xt a reconstruction of data x t to obtain; Output of the data reconstruction x t at a time t; and Controlling the device based on the reconstruction of data x t . [2] Computer-implemented method according to claim 1, wherein the weights α t of the locally linear Kalman filter a function of the switching variable s t are. [3] Computer-implemented method according to claim 2, wherein the weights α t The following relationships hold for the locally linear Kalman filter: At=∑k=1Kαt(k)A(k);αt=(αt(1),....,αt(K))=softmax(st);∑k=1Kαt(k)=1; αt(k)≥0, where A tdenotes a transition matrix that at any given time represents a weighted sum of K transition basis matrices A (k) with 1 ≤ k ≤ K, where the weights α t through the softmax function of the switching variable s t are given. [4] Computer-implemented method according to claim 3, wherein a prediction distribution p(z t |s t ,z t-1 ) for the latent representation z t of the Kalman filter at time t a function N of an average μzt− a latent representation zt− of the Kalman filter before a Kalman update and a covariance ∑zt− the latent representation zt− of the Kalman filter before the Kalman update, which is expressed by p(zt|st,zt−1)=N(μzt−,∑zt−) with μzt−=Atμzt−1+ and ∑zt−=AtΣzt−1+AtT+Iσ, where μzt−1+ a mean value of a previous latent representation zt−1+ of the Kalman filter after a Kalman update, ∑zt−1+ a covariance of the preceding latent representation zt−1+ of the Kalman filter after the Kalman update and I denote an identity matrix. [5] Computer-implemented method according to claim 1, wherein the data x t Time series data and the sensor is an optical sensor, a motor vehicle sensor or an acoustic sensor. [6] Computer-implemented method according to claim 5, wherein the data x t Image data. [7] Computer-implemented method according to claim 6, wherein controlling the device based on the reconstruction comprises controlling an acceleration and a deceleration of the device. [8] Device control system comprising the following: a control unit configured to perform the computer-implemented method according to any one of claims 1 to 7. [9] Device comprising the following: the sensor; and the device control system according to claim 8. [10] Device according to claim 9, wherein the device is a vehicle, a manufacturing machine, a power tool, an automated personal assistant, a monitoring system or a medical imaging system.

Images