learning parameters of a probabilistic model including a gaussian process

Abstract

Parameters of a probabilistic model comprising Gaussian processes are learned. A system (100) is disclosed for learning a set of parameters of a probabilistic model having layers of multiple Gaussian processes (e.g., deep GPs) from a training data set. The set of parameters includes at least induced locations for the multiple Gaussian processes, and parameters of a probability distribution approximating outputs of the multiple Gaussian processes at the multiple induced locations. The probability distribution comprises a multivariate normal probability distribution having a covariance matrix defined by a Kronecker product of a first matrix indicating similarities between the multiple Gaussian processes and a second matrix indicating similarities between the multiple induced locations. A system (200) is also disclosed for determining one or more samples of an output of the probabilistic model for a given input using the set of parameters, e.g., to determine a mean and / or uncertainty estimate for the probabilistic model.

Description

Technical Field

[0001] This invention relates to a system for learning a set of parameters of a probabilistic model comprising a Gaussian process (e.g., a technical system having a complex underlying physical model), and a system for applying the probabilistic model to an input. The invention also relates to a computer-implemented method and a computer-readable medium corresponding to said system, and to a computer-readable medium comprising said set of parameters. Background Technology

[0002] Many technical systems have underlying physical models that are too complex to be explicitly modeled. For example, a model predicting a particular engine emission value might depend in a complex, nonlinear way on its input parameters (such as speed and load) or directly on the input parameters of the engine control unit (ECU). To model the complex nonlinear dependencies that arise in such physical models, Gaussian processes (GPs) can be used as building blocks in probabilistic modeling. Importantly for safety-critical applications, Gaussian processes can be used not only for predictive point estimation but also for providing estimates of associated uncertainties. Models composed of individual GPs are known to be well calibrated; for example, their uncertainty estimates are reasonable. More complex systems can be modeled using so-called deep Gaussian processes, which consist of hierarchical components of GPs.

[0003] In the paper "Doubly Stochastic Variational Inference for Deep Gaussian Processes" by H. Salimbeni and M. Deisenroth, 2017 (accessed via citation in this paper and at https: / / arxiv.org / abs / 1705.08933), an algorithm for training deep GPs based on doubly stochastic variational inference is presented. Based on the mean-field assumption—that the latent processes are posteriorly independent—statistics for each GP are learned independently using an induced point framework. Summary of the Invention

[0004] The inventors recognize that existing techniques for training and using probabilistic models with Gaussian processes have several drawbacks. First, existing techniques for handling such probabilistic models are computationally expensive. For example, a relatively large number of parameters are required to represent Gaussian processes, which means that a large number of iterations are needed to train such models using iterative techniques, and evaluating the models (e.g., calculating the lower bound of their marginal likelihood and the predicted distribution) is costly. Meanwhile, as the inventors recognize, assuming posterior independence implies ignoring the correlations between underlying Gaussian processes. This is detrimental to prediction uncertainty because it leads to an underestimation of the uncertainty of the underlying posterior and the predicted distribution in many cases. This is problematic, especially for safety-critical applications.

[0005] There is a need to improve probabilistic models based on Gaussian processes in order to address these and other problems.

[0006] According to a first aspect of the invention, as defined in claim 1, a system for learning a set of parameters for a probabilistic model is provided. According to another aspect of the invention, as defined in claim 10, a system for applying a probabilistic model to an input is provided. According to another aspect of the invention, as defined in claim 12, a method for learning a set of parameters for a probabilistic model is provided. According to another aspect of the invention, as defined in claim 13, a method for applying a probabilistic model to an input is provided. According to another aspect of the invention, as defined in claims 14 and 15, a computer-readable medium is provided.

[0007] Using these methods, the inventors were able to develop a family of probability models with various advantages.

[0008] Using multiple Gaussian processes allows for relatively accurate models of various types of functions, such as systems with complex nonlinear dependencies (e.g., models of technological systems or physical phenomena). By using the Kronecker product to constrain the covariance matrix of a multivariate normal probability distribution for a set of Gaussian processes, the correlations between Gaussian processes can be captured in the model, particularly through the first matrix of the Kronecker product. Thus, various multidimensional functions can be modeled more accurately, and in particular, uncertainties attributable to posterior correlations between Gaussian processes can be taken into account. Indeed, the inventors have found that assuming posterior independence between latent processes can lead to underestimated predictive uncertainties, especially if some latent processes perform interpolation while others perform extrapolation. Using the Kronecker product in the approximate posterior allows for modeling of dependencies between Gaussian processes, and if independence is assumed, it helps reduce inaccuracies.

[0009] Furthermore, various methods allow for more efficient learning of the parameter set of probabilistic models and / or application of probabilistic models to inputs. Due to the Kronecker structure, relatively rich models are obtained using relatively few parameters. Surprisingly, the number of parameters is reduced not only compared to models with a full covariance matrix having a multivariate normal distribution, but also even compared to models with independent covariance matrices for the corresponding Gaussian processes not constrained by a common Kronecker structure.

[0010] Improved storage efficiency and reduced overfitting risk are achieved by using fewer parameters than for independent covariance matrices or a complete covariance matrix. Furthermore, the Kronecker decomposition leads to more efficient optimization. First, this is because a smaller number of iterations is sufficient to achieve convergence to the optimal parameter set. Second, parameterizing from the multivariate normal probability distribution and various probability distributions derived therefrom using the first and second matrices instead of the covariance matrix makes various computations more efficient. For example, calculating the lower bound of the marginal likelihood and / or sampling from the approximate predictive distribution of the probabilistic model can be more efficient. Since the multivariate normal probability distribution is typically used for both optimizing the parameter set and sampling the output of the probabilistic model, using the Kronecker decomposition of the covariance matrix improves performance in both cases.

[0011] The above aspects of the present invention can relate to a probabilistic model, sometimes also called a Bayesian model. Generally, given an input, evaluating a probabilistic model allows sampling (e.g., probabilistically sampling) of the output corresponding to the input, and / or providing a probability distribution for the output corresponding to the input. The input is typically a vector comprising one or more values. In some embodiments, the output is a value; in other embodiments, the output comprises multiple values. A probabilistic model typically includes one or more random variables, such as latent variables and / or output variables. Latent variables can be defined by a probability distribution or by a stochastic process. They can be instantiated based on one or more of the current input to the probabilistic model, sampled values ​​of other random variables, and / or parameters (e.g., hyperparameters) of the probabilistic model. A typical example of a stochastic process is a Gaussian process. Various examples of probabilistic models are provided below.

[0012] Probabilistic models are typically discriminative models, for example, models used to predict one or more output variables based on one or more input variables. In other words, probabilistic models can be used as approximations of functions. In various embodiments, this function captures the behavior of a technological system, such as a computer-controlled machine, like a robot, vehicle, home appliance, power tool, manufacturing machine, personal assistant, or access control system. In embodiments, probabilistic models are used to predict one or more physical output quantities based on at least one or more physical input quantities (e.g., obtained from measurements or set by a control unit). For example, measurements can be obtained from one or more of various sensors (e.g., image or video cameras, radar, lidar, or ultrasonic sensors). For example, predictions can be used to control the technological system whose performance is measured. Interestingly, probabilistic models can allow for approximations of the output of such functions, even when the underlying model (e.g., a physical model of the technological system's behavior) is unknown and / or too complex to be explicitly modeled.

[0013] In one embodiment, the probabilistic model is used to predict engine emissions based on speed and load or corresponding ECU parameters. In other embodiments, the probabilistic model may be an inverse dynamics model, such as a model that calculates torque based on inputs including one or more of position, velocity, and acceleration. For example, such an inverse dynamics model can calculate the torque of a robot. In yet another embodiment, the probabilistic model may be a model that calculates disease risk for a patient based on medical data (e.g., physiological measurements and / or data from patient records).

[0014] Various embodiments relate to probabilistic models comprising or composed of multiple Gaussian processes. In the field of machine learning, Gaussian processes are known as a tool for providing output predictions (typically real numbers) for an input (typically a vector containing multiple values, such as real numbers). Predictions can include not only values ​​for the output but also estimates of uncertainty, as discussed in more detail below. Conceptually, a Gaussian process can provide an estimate of the output value for the current input based on known output values ​​for other inputs similar to the current input, and / or make its uncertainty estimate based on several other similar inputs and their similarity. Gaussian processes are typically specified by a prior mean function and a covariance function (also called a kernel). In embodiments, the prior mean function is a zero function. In embodiments, the kernel is non-degenerate. The mean function and / or kernel can be limited by a set of hyperparameters. Gaussian processes may be particularly suitable as building blocks for modeling complex nonlinear dependencies (e.g., in models of technical systems or physical phenomena).

[0015] In various embodiments, the probabilistic model is organized in one or more layers. For example, the number of layers may be at most or at least two, at most or at least five, or at most or at least ten. One of the one or more layers includes multiple Gaussian processes, for example, at most or at least two, at most or at least ten, or at most or at least twenty. In other words, the multiple Gaussian processes have a common or at least partially overlapping set of inputs. For example, the inputs may include the outputs of one or more other Gaussian processes and / or the inputs of the probabilistic model. Such models are sometimes referred to as multi-GP models. In some embodiments, the probabilistic model may consist of a single layer comprising multiple Gaussian processes. Such Gaussian processes may be combined by weighted summation; in other words, if the output is one-dimensional, the probabilistic model may be a summation of GP models, or if the output is a vector, the probabilistic model may be a multi-output Gaussian process. In other embodiments, the probabilistic model includes multiple layers. For example, each layer may include one or more Gaussian processes; for example, the probabilistic model may be a so-called deep Gaussian process. Typically, the output of a probabilistic model is noisy; for example, noise such as Gaussian noise is added to the output of a Gaussian process or other random variables to obtain the corresponding output of the probabilistic model.

[0016] Probabilistic models can also include random variables that are not generated by the Gaussian process, at the same layer as the Gaussian process and / or at other layers. For example, a probabilistic model may include random variables that provide non-Gaussian noise, such as Laplace noise or exponential noise. Probabilistic models may also include discrete random variables (e.g., based on Bernoulli or Poisson distributions). As another example, a probabilistic model may include latent variables representing nodes in a neural network, such as latent variables... ,in h It is an activation function. B These are weights (e.g., in the case of a Gaussian prior), and X This is the input. Various other non-Gaussian processes can also be readily incorporated into the probabilistic model. Generally, given an input, samples of the output corresponding to the input can be obtained by sampling the values ​​of the random numbers corresponding to the inputs of the probabilistic model, and / or the probability distribution of the output corresponding to the input can be obtained by marginalizing the latent variables of the probabilistic model.

[0017] Typically, the probabilistic model structure, such as the number of layers, the number / type of stochastic processes at each layer, the type of kernel function for the Gaussian process, etc., is predefined. However, one or more of these structural aspects of the probabilistic model can also be automatically determined and / or stored as parameters of the probabilistic model for use, for example, by using various meta-learning techniques known in the art during training.

[0018] The above aspects of the present invention relate to a set of parameters for a probabilistic model. For example, if noise is added to the output of a probabilistic model according to a probability distribution, the parameters may include one or more noise parameters describing the noise model, such as variance or standard deviation for additive Gaussian noise. Parameters may also include various hyperparameters, such as hyperparameters of a specific probability distribution / stochastic process of the probabilistic model and / or global hyperparameters (e.g., learning rate, number of nodes or layers, etc.).

[0019] The parameter set can be learned based on a training dataset; for example, this learning can be supervised and used when applying the trained model to a given input. In other words, the parameter set can be determined by optimizing the parameter set relative to a training dataset, for example, comprising multiple training inputs and corresponding training outputs. Typically, the training inputs are vectors, such as real-valued vectors; the training outputs can be, for example, real-valued numbers, but can also be vectors. For example, the parameter set can be selected during training based on the ability to reproduce the training output given the training inputs and / or to generalize from the training dataset to new test points as much as possible. When the model is applied, the training dataset is typically no longer needed; however, embodiments where the probabilistic model involves lazy learning are possible, where at least a portion of the training dataset is used when the model is applied. The parameter set can be optimized using various techniques, such as stochastic gradient descent, simulated annealing, or various Monte Carlo-based methods. As is common in machine learning, obtaining exact optimal values ​​is often difficult or impossible; for example, optimization can be heuristic and / or the obtained set of parameters may correspond to local optima.

[0020] In various embodiments, at least some of the values ​​included in the training input and output are obtained as sensor measurements. In use, the model can then also be applied to inputs that include sensor measurements.

[0021] Specifically, the parameter set may include one or more parameters of a plurality of Gaussian processes whose outputs are approximated. In various embodiments, the parameter set includes a set of induced positions for the plurality of Gaussian processes. In other words, the plurality of Gaussian processes share a common set of induced positions. An induced position can be a point at which the plurality of Gaussian processes can be evaluated, for example, a vector equal to the values ​​of several inputs of the plurality of Gaussian processes, such as several Gaussian processes at previous layers in the case of a deep Gaussian process model. Induced positions are sometimes also referred to as induced points, the location of induced points, or pseudo-inputs. Conceptually, optimizing the parameter set may include selecting a set of induced points that allows for the optimal reproduction of the training output given a training input; in other words, selecting a set of induced points that represent the inputs of the Gaussian processes (e.g., the training inputs or outputs of previous layers). Induced points generally do not correspond to or are derived from the training inputs, although in some embodiments, induced points may correspond to a subset of the training inputs or be derived from a subset of the training inputs using techniques known in the art.

[0022] Interestingly, the parameter set can also include parameters for a probability distribution that approximates the outputs of multiple Gaussian processes at multiple induction locations; for example, this probability distribution could be an approximate posterior distribution over such outputs. The outputs are sometimes referred to as pseudo-outputs. When a probabilistic model is applied to an input, the approximate posterior and / or approximate outputs of multiple Gaussian processes at induction locations can be used to determine the outputs of multiple Gaussian processes for the input. The probability distribution for the outputs of multiple Gaussian processes at multiple induction locations can include a multivariate normal distribution. In other words, a sample of a multivariate normal distribution can represent the output of each Gaussian process for each induction location. The use of a multivariate normal distribution allows for capturing correlations between multiple Gaussian processes; for example, it is generally not assumed that multiple Gaussian processes are posteriorly independent of each other—this would be the case, for example, when each Gaussian process is modeled by a separate probability distribution. The parameters of a multivariate normal distribution are typically parameterized, including its covariance matrix; the parameters can also be parameterized for their mean.

[0023] The general covariance matrix for the probability distribution of the output of a Gaussian process can be relatively large; however, it is of interest that the covariance matrix can be defined by the Kronecker product of a first matrix indicating the similarity between multiple Gaussian processes and a second matrix indicating the similarity between multiple induced positions. In other words, the set of parameters may include entries of the first and / or second matrices, or otherwise define the first and / or second matrices, rather than directly including all entries of the covariance matrix. Typically, the first and second matrices are positive definite real matrices. The first matrix may be a square with a size equal to the number of Gaussian processes, and the second matrix may be a square with a size equal to the number of induced positions.

[0024] In addition to the induced location and probability distribution parameters, the parameter set may also include a set of hyperparameters for the prior mean function and / or kernel of multiple Gaussian processes. Typically, each of the multiple Gaussian processes is defined by the same set of hyperparameters, although this is not strictly necessary. As discussed above, in various embodiments, the probabilistic model includes multiple sets of Gaussian processes; for example, the probabilistic model may include a first set of multiple Gaussian processes at a first layer and a second set of multiple Gaussian processes at a second layer different from the first layer. Alternatively or additionally, the probabilistic model may include additional sets of multiple Gaussian processes at the first layer. However, the sets of multiple Gaussian processes discussed herein are included in the same layer. Typically, different sets of Gaussian processes are defined by different parameters; for example, each set of Gaussian processes may have separate kernel hyperparameters, separate sets of induced locations, and / or separate parameters that approximate the probability distribution of its output at the respective induced locations; for example, each set of multiple Gaussian processes may have separate first and second matrices. However, some parameters, particularly kernel hyperparameters, may also be shared. Sharing parameters among multiple sets of Gaussian parameters represents a trade-off between minimizing the number of parameters and improving the expressiveness of the probabilistic model, where minimizing the number of parameters can be improved, for example, by reducing the number of iterations required for optimization.

[0025] Optionally, optimizing the parameter set includes maximizing a lower bound on evidence relative to the training dataset. For example, the lower bound on evidence can be optimized by maximizing the likelihood of the observed training outputs under the output distribution of a probabilistic model conditioned on the training inputs. The lower bound on evidence may also include a regularization term that penalizes approximate posteriors that are too far from their corresponding priors above the output, for example, to prevent overfitting due to excessively high model complexity, as discussed further below. The lower bound on evidence can be an approximate lower bound on evidence, for example, approximated using Monte Carlo sampling. By maximizing the lower bound on evidence, a parameter set can be obtained that optimally attempts to reproduce and / or inductively approximate the function to be modeled from the training dataset.

[0026] Optionally, optimizing the parameter set involves sampling the output of the probabilistic model given the training inputs and computing the likelihood of obtaining the training output corresponding to the training inputs from the sampled outputs under a noise distribution. For example, the probabilistic model can model noisy outputs (e.g., outputs to which noise is added according to a noise distribution). The likelihood of obtaining the training outputs from the sampled outputs can indicate the accuracy of the probabilistic model with respect to the training inputs and can be used, for example, in computing a lower bound of evidence or another objective function to be optimized. By sampling as described above, for example using a Kronecker structure for multiple Gaussian processes of the probabilistic model, this likelihood can be computed relatively efficiently and / or the neglect of the correlation between Gaussian processes can be avoided.

[0027] Optionally, sampling the output of the probabilistic model for the training input includes: determining a sample probability distribution of the output of the multiple Gaussian processes for the training input by marginalizing the probability distribution of the outputs of the multiple Gaussian processes at multiple induced locations from the joint probability distribution; and sampling the outputs of the multiple Gaussian processes for the training input based on the sample probability distribution. Sampling is typically performed layer by layer; for example, the sample probability distribution of the outputs of the multiple Gaussian processes can be determined based on the sampled outputs of previous layers of the probabilistic model. For example, the joint probability can include at least the outputs at multiple induced points and the output at the training input. In other words, marginalization is performed above the induced points in order to sample the output. For example, it is not necessary to sample the outputs at the induced points, which can improve the efficiency of the sampling process and can lead to a more accurate model because inaccuracies attributable to sampling the induced outputs are avoided.

[0028] Optionally, maximizing the lower bound of evidence involves performing stochastic gradient descent on the objective function that includes the lower bound of evidence. In machine learning, stochastic gradient descent is known itself as a general and efficient optimization method. Of particular interest is that probabilistic models, as defined above, are highly susceptible to stochastic gradient descent. For example, the objective function (e.g., the lower bound of evidence) generated by the probabilistic model typically decomposes between training inputs and is differentiable. In embodiments, learning rates for different parameters from the parameter set can be adaptively computed to improve learning; for example, stochastic gradient descent may include adaptive moment estimation. Stochastic gradient descent can be particularly helpful for large datasets; for smaller datasets, non-stochastic optimization methods can also be used.

[0029] Optionally, determining the parameter set includes optimizing the Kullback-Leibler divergence between the probability distributions approximating the outputs of multiple Gaussian processes at multiple induced locations and the prior probability distributions of the outputs of multiple Gaussian processes at multiple induced locations. For example, the Kullback-Leibler divergence might appear in the calculation of the lower bound of evidence to be maximized, but it can also be used in various other objective functions to be minimized or maximized. In calculating the Kullback-Leibler divergence, the Kronecker structure, which can be approximated by the covariance matrix, is used efficiently, as described in detail elsewhere; the Kullback-Leibler divergence can be calculated without explicitly storing and / or calculating the covariance matrix defined by the Kronecker product. Therefore, using the Kronecker structure makes the optimization more efficient.

[0030] Optionally, the first matrix is ​​a diagonal matrix. By taking the off-diagonal elements of the first matrix to zero, the number of parameters can be further reduced, leading to even more efficient learning and use, and even less storage. This is especially true in settings where the approximate independence of at least several Gaussian processes can be assumed, or where they never enter the interpolation / extrapolation scenario. This results in, for example, a more resource-efficient but still sufficiently expressive model compared to having separate probability distributions with separate covariance matrices for each of the multiple Gaussian processes. On the other hand, in settings where it is necessary to capture the similarity between different Gaussian processes in the first matrix, the first matrix can have at least one non-zero diagonal element that allows such similarity to be captured, leading to a more expressive model.

[0031] Optionally, one Gaussian process in a plurality of Gaussian processes has a quadratic exponential kernel. In particular, as discussed elsewhere, each Gaussian process in a plurality of Gaussian processes can have a quadratic exponential kernel, for example, defined by the same set of hyperparameters. Quadratic exponential kernels have various advantageous properties, such as being a universal kernel, having good differentiability, etc. In this embodiment, the quadratic exponential kernel has autocorrelation determination. Besides the quadratic exponential kernel, various other types of non-degenerate kernels can be used, such as kernels that give the covariance function full rank, such as rational quadratic kernels or various types of periodic kernels. Generally, such kernels cannot be written as a finite sum of the functional products of their first and second arguments. As discussed, different sets of multiple Gaussian processes in a probabilistic model can have different kernels. For example, each set of multiple Gaussian processes can have a quadratic exponential kernel that may be defined by a separate set of hyperparameters, or some sets of Gaussian processes can have quadratic exponential kernels while other Gaussian processes have other kernels. In this embodiment, each Gaussian process in the probabilistic model has a non-degenerate kernel.

[0032] Optionally, the parameter set of the probabilistic model includes one or more kernel hyperparameters, and each Gaussian process in the plurality of Gaussian processes has a kernel defined by the one or more kernel hyperparameters. In other words, each Gaussian process in the plurality of Gaussian processes can be defined by the same set of hyperparameters. Such a common set of hyperparameters can help ensure the accuracy of the Kronecker approximation. Having common hyperparameters can further improve performance by limiting the total number of parameters and / or reducing the computational complexity of the resulting calculations.

[0033] Alternatively, when applying a probabilistic model to the input, multiple samples of the probabilistic model's output with respect to the input can be determined, thereby allowing for the determination of an uncertainty estimate. The uncertainty estimate can be, for example, a statistical parameter such as variance or standard deviation, or a visual uncertainty estimate such as a sample histogram. As discussed above, particularly accurate uncertainty estimates are obtained because the probabilistic model can capture the correlations between multiple Gaussian processes; furthermore, efficiency can be improved by utilizing the particularly efficient sampling based on the Kronecker structure of the covariance matrix, as also discussed elsewhere, to obtain the uncertainty estimate from multiple samples.

[0034] Those skilled in the art will appreciate that two or more of the above-mentioned embodiments, implementations and / or alternative aspects of the present invention can be combined in any manner deemed useful.

[0035] Any computer-implemented method and / or any computer-readable medium may be modified and varied by those skilled in the art based on this description, the modifications and variations corresponding to the modifications and variations described for the corresponding system. Attached Figure Description

[0036] These and other aspects of the invention will become clear and further illustrated by reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, in which...

[0037] Figure 1 A system for learning a set of parameters for a probabilistic model by optimizing the parameters relative to the training dataset is shown.

[0038] Figure 2 This illustrates a system for applying a probability model to an input;

[0039] Figure 3 A detailed example of a probabilistic model comprising one or more layers is shown, where each layer comprises multiple Gaussian processes, and in this case, deep Gaussian processes;

[0040] Figure 4 A detailed example is shown of how the parameters of a probabilistic model can be determined and used to apply the probabilistic model to an input, the parameters of which include the induced position and first and second matrices defining the covariance matrix for approximating the output of the Gaussian process at the induced position.

[0041] Figure 5 A method is shown to learn the set of parameters of a probabilistic model by optimizing the parameters relative to the training dataset;

[0042] Figure 6 This demonstrates a method for applying a probabilistic model to the input;

[0043] Figure 7 A computer-readable medium including data is shown;

[0044] Figures 8a-8d The performance improvements of the proposed probabilistic model on various benchmark datasets are shown.

[0045] It should be noted that the figures are purely schematic and not drawn to scale. In each figure, elements corresponding to those already described may have the same reference numerals. Detailed Implementation

[0046] Figure 1 A system 100 for learning a set of parameters for a probabilistic model is illustrated. The probabilistic model may include one or more layers. One of the layers may include multiple Gaussian processes. System 100 may include a data interface 120 and a processor subsystem 140, which may communicate internally via data communication 124. The processor subsystem 140 may be configured to access a training dataset 030, including multiple training inputs and corresponding training outputs, during operation of system 100 and by using the data interface 120. For example, as Figure 1 As shown, data interface 120 can provide access 122 to external data storage device 020, which may include the data 030. Alternatively, data 030 may be accessed from an internal data storage device that is part of system 100. Alternatively, data 030 may be received from another entity via a network. Generally, data interface 120 can take various forms, such as a network interface to a local area network or wide area network (e.g., the Internet), a storage interface to an internal or external data storage device, etc. Data storage device 020 can take any known and suitable form.

[0047] Processor subsystem 140 may also be arranged to determine a parameter set 050 of a probabilistic model during operation of system 100 by optimizing a parameter set 050 relative to training dataset 030. Parameter set 050 may include a set of induced positions for multiple Gaussian processes. Parameter set 050 may also include parameters approximating a probability distribution of the outputs of the multiple Gaussian processes at the multiple induced positions. The probability distribution may include a multivariate normal probability distribution. The covariance matrix of the multivariate normal distribution may be defined by a Kronecker product of a first matrix indicating similarity between the multiple Gaussian processes and a second matrix indicating similarity between the multiple induced positions. Optionally, processor subsystem 140 may be configured to, for example, store the determined parameter set 050 using data interface 120, and / or otherwise provide the determined parameter set 050 to a system for applying the model to inputs.

[0048] Reference Figure 3The various details and aspects of the operation of System 100 are further elaborated, including its optional aspects.

[0049] Generally, System 100 may be embodied as or within a single device or apparatus, such as a workstation (e.g., a laptop or desktop-based) or a server. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such CPUs and / or graphics processing units (GPUs) and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory like RAM or non-volatile memory like flash memory. Alternatively, the functional units of the system (e.g., data interfaces and processor subsystems) may be implemented in the device or apparatus as programmable logic, such as a field-programmable gate array (FPGA). Generally, each functional unit of the system may be implemented as a circuit. Note that System 100 may also be implemented in a distributed manner, for example involving different devices or apparatuses such as distributed servers, for example in the form of cloud computing.

[0050] Figure 2 A system 200 for applying a probabilistic model to an input is illustrated. The probabilistic model may include one or more layers. One of the layers may include multiple Gaussian processes. System 200 may include a data interface 220 and a processor subsystem 240, which may communicate internally via data communication 224. The processor subsystem 240 may be configured to access the input 060 and parameter set 050 of the probabilistic model during operation of system 200 and by using the data interface 220. Parameter set 050 may include a set of multiple induced positions for the multiple Gaussian processes. Parameter set 050 may also include parameters approximating the probability distribution of the outputs of the multiple Gaussian processes at the multiple induced positions. The probability distribution may include a multivariate normal probability distribution. The covariance matrix of the multivariate normal distribution may be defined by the Kronecker product of a first matrix indicating similarity between the multiple Gaussian processes and a second matrix indicating similarity between induced positions.

[0051] For example, such as Figure 2As shown, data interface 220 can provide access to external data storage device 022, which may include the data 050, 060. Alternatively, data 050, 060 can be accessed from an internal data storage device that is part of system 200. Alternatively, data 050, 060 can be received from another entity via a network. Generally, data interface 220 can take various forms, such as a network interface to a local area network or wide area network (e.g., the Internet), a storage interface to an internal or external data storage device, etc. Data storage device 022 can take any known and suitable form.

[0052] The processor subsystem 240 may also be arranged to determine, during operation of system 100, an output sample of the probability model with respect to input 060 based on a parameter set 050 of the probability model. As an optional component, system 200 may include a display output interface (not shown) or any other type of output interface for outputting samples to a presentation device such as a display. For example, the display output interface may generate display data for the display, which causes the display to present the sample and / or its uncertainty estimate in a sensor-perceptible manner. As an optional component, system 200 may include one or more sensor interfaces for obtaining at least a portion of input 060 from one or more sensors. As an optional component, system 200 may include a control interface for controlling a technical system, such as a car engine, based on the output of the probability model.

[0053] Reference Figure 3 This section further elaborates on the various details and aspects of the operation of System 200, including its optional aspects.

[0054] Generally, system 200 may be embodied as or within a single device or apparatus, such as a workstation (e.g., a laptop or desktop-based) or a server. The device or apparatus may include one or more microprocessors executing appropriate software. For example, a processor subsystem may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such CPUs and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory like RAM or non-volatile memory like flash memory. Alternatively, the functional units of the system (e.g., data interfaces and processor subsystems) may be implemented in the device or apparatus as programmable logic, such as a field-programmable gate array (FPGA). Generally, each functional unit of the system may be implemented as a circuit. Note that system 200 may also be implemented in a distributed manner, for example involving different devices or apparatuses such as distributed servers, for example in the form of cloud computing.

[0055] Figure 3 A detailed, but non-limiting, example of a probabilistic model comprising one or more layers is shown, wherein one of the layers comprises multiple Gaussian processes. In this case, the probabilistic model 300 is a deep GP or a deep Gaussian process, such as a multi-GP model with multiple layers.

[0056] Generally, various embodiments may involve... n A probabilistic model consisting of potential Gaussian processes (GPs) is denoted as follows: The parameter set for learning a probabilistic model can include parameters optimized relative to a training dataset that includes multiple training inputs and corresponding training outputs. For example, a task can be viewed as being based on training inputs... and typical noisy training output of N An observation Let's learn functions In the case of multidimensional output, each You can specify only Some of the output components. For example, a typical deep Gaussian process can include 2 to 5 layers, for example, each layer has 10-dimensional input and 10 Gaussian processes.

[0057] Probabilistic models can place a single GP prior on one or more (e.g., each) of the latent functions. ,have The prior mean function and The kernel, typically a positive definite kernel. In this example, n Each prior GP is considered to be independent of the others, but as those skilled in the art understand, more general settings can also be handled, for example, coupled priors can be taken on multiple nodes.

[0058] In various embodiments, the parameter set may include the induced position for the Gaussian process. For example, some or all of the GP may be equipped with M A induced input ,in .

[0059] As shown for depth GP 300, each layer It can include A Gaussian process. For example, the figure shows the first layer 320, which includes a Gaussian process. 321 until 322. The figure also shows the last one. L Layer 330, which includes Gaussian processes 331 until 332. Total number of Gaussian processes with depth GP 300n It can be calculated as .

[0060] In the layer l The input to the Gaussian process at this point can include the Gaussian process of the previous layer. The output of, for example, a Gaussian process. It can represent a mapping For example, the model can be fully connected at this layer. However, a Gaussian process can also map only a subset of the values ​​from previous layers to the output. The first layer can have inputs... 310. Mathematically, the relationship between the layers can be represented as: .

[0061] The Gaussian process 300 shown in the figure has a single output. y 340 (e.g.) The single output y 340 is obtained by adding Gaussian independent and identically distributed (iid) noise to the output of layer 330, for example... .

[0062] Mathematically, the enhanced joint probability using the induced point framework for a deep Gaussian process 300 can be expressed as:

[0063]

[0064] in The label indicates the training output corresponding to the corresponding training input. Indicating probabilistic models in layers l The output at the training input. Indicating probabilistic models in layers l The output at the induced position.

[0065] vector and It is obtained by stacking the corresponding outputs of each layer.

[0066] In various embodiments, the regularity of the model can be increased by assuming that the underlying processes in the same layer have the same structure (e.g., the same induction points and the same kernel hyperparameters). For example, the following relationship can be assumed to hold:

[0067] ,

[0068]

[0069] here , , and , Similar. From mean Covariance Matrix It can be limited according to the following relationship:

[0070]

[0071] The posterior of a deep Gaussian process 300 can be approximated by variational inference using the following family of variational inferences:

[0072]

[0073] In various embodiments, a stochastic lower bound on the marginal likelihood is obtained using variational inference techniques such as dual stochastic variational inference.

[0074] although Figure 3 A depth GP 300 including Gaussian processes 321, 322, 331, and 332 is shown, but those skilled in the art will understand that the above technical quantities (e.g., posterior to the probabilistic model) can be generalized in various ways. For example, the probabilistic model may include various other latent variables not generated by Gaussian processes, and multiple sets of Gaussian processes may be included at a single layer, for example, with different hyperparameters for each set, and so on. On the other hand, the mathematical description of a probabilistic model with a single layer of Gaussian processes can also be achieved by setting... And this can be deduced from the above description.

[0075] Figure 4 Detailed, but non-limiting, examples are shown of how the parameters of a probability model can be determined and / or used to apply the probability model to an input.

[0076] Figure 4 The diagram shows a probabilistic model PM 440, which comprises one or more layers. One of the layers comprises multiple Gaussian processes. For illustrative purposes, three Gaussian processes 441-443 are shown in the diagram, but this is not a limitation. For example, the probabilistic model PM may correspond to or include a deep Gaussian process 300.

[0077] The figure also shows the parameter set PAR 450 of the probabilistic model PM. The parameter set PAR can be learned, for example, by system 100 based on a training dataset including training input TI 431 and training output TO 432, as discussed in more detail later. The parameter set PAR can also be used to apply the probabilistic model PM to input I 470 in order to obtain sampled output SO 480, for example, by system 200, as discussed in more detail later.

[0078] For multiple Gaussian processes 441-443, the parameter set PAR can include a set of multiple induced locations IL 451. (This refers to the set of parameters for multiple Gaussian processes 441-443.) of M The set of induced positions can be denoted as Typically, multiple Gaussian processes 441-443 share a common set of induced points, but Gaussian processes at different layers typically have different sets of induced points, and it is also possible, for example, for different sets of Gaussian processes at the same layer to have different sets of induced points. For example, the number of induced points for a set of multiple Gaussian processes could be 100.

[0079] The parameter set PAR can also include the outputs of multiple Gaussian processes at multiple induced locations. The parameters of the approximate probability distribution. For example, such probability distributions may arise when using variational inference to approximate the posterior of a probabilistic model PM, as illustrated above. Of interest, the probability distributions approximating the outputs of multiple Gaussian processes at multiple induction sites can include a multivariate normal probability distribution, where the covariance matrix of the multivariate normal distribution is defined by the Kronecker product of a first matrix FM452 and a second matrix SM453. The first matrix FM452 indicates the similarity among the multiple Gaussian processes 441–443, and the second matrix SM453 indicates the similarity among the multiple induction sites IL. For example, the first matrix FM could be of a size equal to the number of Gaussian processes. The square matrix, and the second matrix SM second matrix The size can be equal to the number of induced points. M A square formation.

[0080] In mathematics, the first matrix Second matrix The following Kronecker factor decomposition can be formed to approximate the probability distribution:

[0081]

[0082] Multivariate normal probability distribution mean First matrix and / or the second matrix These can be free variational parameters. In other words, the parameter set PM can include the first matrix FM, the second matrix SM, and / or the mean, which define the multivariate normal probability distribution. .

[0083] In various embodiments, the first matrix FM can be a diagonal matrix; for example, the diagonal elements of the first matrix FM are free parameters, where the off-diagonal elements of the first matrix are assumed to be zero. As discussed above, this allows for a reduction in the number of parameters. In other embodiments, the first matrix FM has at least one non-zero diagonal element, thereby allowing such relationships between different Gaussian processes to be captured and effectively utilized when applying the model PM.

[0084] The parameter set PAR can include various other parameters, such as those corresponding to other parts of the probabilistic model PM (e.g., other latent processes of the probabilistic model). In particular, various probabilistic models use noisy outputs; for example, the output 340 of a deep Gaussian process 300 can be noisy. In such cases, the parameter set can include one or more parameters relating to the noise distribution of the output of the probabilistic model PM, such as the standard deviation of a normal distribution. Or variance, etc.

[0085] The parameter set PAR may also include one or more hyperparameters for the multiple Gaussian processes 441-443. For example, the parameter set may include hyperparameters for the prior mean function of the multiple Gaussian processes 441-443 and / or kernel hyperparameters KH 454 for the kernel function of the multiple Gaussian processes. In various embodiments, the multiple Gaussian processes share at least some hyperparameters; for example, each Gaussian process in the multiple Gaussian processes may have a kernel defined by one or more kernel hyperparameters KH. Gaussian processes at other layers of the probabilistic model PM, or other sets of Gaussian processes at the same layers as Gaussian processes 441-443, may have different types of prior means and kernel functions, and / or different hyperparameters for their functions, although it is also possible to use the same set of hyperparameters for all Gaussian processes of the probabilistic model.

[0086] Various types of kernels can be used for Gaussian processes, with correspondingly varying sets of kernel hyperparameters KH. The kernels used are typically non-degenerate. For example, a quadratic exponential kernel, such as one with autocorrelation determination, can be used. Mathematically, given a vector... This type of kernel can be limited to

[0087] ,

[0088] The kernel hyperparameter KH includes parameters specific to the input dimension that indicate the relevance of the corresponding input features. D Length scaling hyperparameter and global, typically non-negative scaling parameters. .

[0089] In various embodiments, the prior mean function for a Gaussian process can be set to zero; for example, in this case, hyperparameters used to define the prior mean function may not be required. In other cases, a prior mean function with hyperparameters, such as a polynomial prior mean function, can be used, for example, whose coefficients are learned as hyperparameters.

[0090] As mentioned, the parameter set PAR can be determined (e.g., learned) based on a training dataset that includes multiple training inputs TI and training outputs TO. Training may include optimizing the parameter set PAR relative to the training dataset, such optimization being performed, for example, by selecting the parameter set PAR such that the probabilistic model PM is optimally inductively derived from the training inputs TI and training outputs TO to new test points.

[0091] In various embodiments, optimizing the parameter set PAR involves maximizing a lower bound on evidence relative to the training dataset. The use of a lower bound on evidence is not necessary: ​​other ways of capturing the correspondence between the probabilistic model PM and the training dataset, such as measuring the difference between the output of the probabilistic model PM for training input TI and training output TO, are also possible. Since precise computation of the lower bound on evidence or other optimization objectives may be infeasible and / or computationally expensive, in various embodiments, the quantity to be optimized (e.g., the lower bound on evidence) is a stochastic approximation. In any case, the quantity to be optimized can be captured as the objective function to be optimized using known optimization techniques. For example, in Figure 4 In the example shown, stochastic gradient descent SGD 460 is performed on an objective function that is equal to the lower bound of evidence in this case. Known optimization frameworks can be used, such as implementations of stochastic gradient descent based on tensor flow.

[0092] For example, in mathematics, such as Figure 3 The lower bound of evidence for the deep Gaussian process shown can be derived using double stochastic variational inference as follows:

[0093]

[0094] in , As discussed above, and

[0095]

[0096] It is the edge of the last layer.

[0097] Note that the lower bound of evidence for this type of probabilistic model includes terms that can be approximated as follows: : Training inputs for a given set TI In the case of one or more outputs of the probabilistic model PM Sampling is performed, and the noise distribution (in this case, the output) is calculated. y Obtain the corresponding training input from the sampled output under a noise distribution of 340. The training output of the set TO Likelihood It can be executed layer by layer. Sampling, for example, the sampled output of a previous layer can be used to sample at the current layer, thereby avoiding the analysis of the model as a whole. As those skilled in the art will understand, such terms also generally arise for other types of probabilistic models (PMs) that include multiple Gaussian processes.

[0098] Specifically, in order to sample the output STO 433 of the probabilistic model for the training input of the set TI, it can be done by sampling from the joint probability distribution (e.g. In this process, the probability distribution in the layer is marginalized to approximate the output of multiple Gaussian processes at multiple induced locations. This is used to determine the sample probability distribution of the outputs of multiple Gaussian processes for the training input. Then, the layer outputs of the multiple Gaussian processes for the training input can be sampled according to the sample probability distribution. Interestingly, the Kronecker product appearing in the probabilistic model PM can lead to an expression based on the first matrix FM and the second matrix SM, which can be used to perform such sampling particularly efficiently, for example, without explicitly storing or computing their Kronecker products. For example, continuing the above derivation of the deep Gaussian process 300, the variational posterior in the induced output... The above marginalization can be performed as follows:

[0099] .

[0100] For example, a multivariate normal distribution with this mean and covariance can be obtained. Samples from the approximate predicted distribution can be obtained, for example, by replacing the above training dataset with a single training input TI or input I. Therefore, sampling the outputs of multiple Gaussian processes, as illustrated here, can include determining the covariance matrix based on the Kronecker product of functions of the first and second matrices. The Kronecker product of the first and second matrices is not required.

[0101] As can be observed in this example, the marginal Depends only on the corresponding input For example, the lower bound can be decomposed among the training inputs TI. As a result of modeling the outputs of multiple Gaussian processes using a multivariate normal probability distribution, in this example, the function values ​​of the same layer... , They are coupled. Therefore, the outputs of multiple Gaussian processes can be sampled according to the joint probability distribution. Furthermore, it can be observed that determining the above probability distribution involves inverting the single probability distribution for groups 341-343 of Gaussian processes. matrix Instead of, for example, in a technique where Gaussian processes are not modeled by a joint multivariate normal distribution as in probabilistic models like PM, the individual matrices for each Gaussian process are inverted, which represents a significant computational saving.

[0102] Various optimizations used to determine the parameter set PAR can also involve optimizing the probability distribution of the outputs of multiple Gaussian processes at multiple induced locations (e.g., ) and the probability distribution of the outputs of multiple Gaussian processes at multiple induced locations (e.g., The Kourbach-Leibler divergence between (e.g., the Kourbach-Leibler divergence). For example, this type of Kourbach-Leibler divergence may occur at the lower bound of evidence (such as the bound derived above). While it can be used in other optimization objectives to capture the difference between approximate posterior and prior probability distributions, it can also be applied in other optimization objectives. Interestingly, parameterized PAR based on the Kronecker product allows for efficient determination of the Körbach-Leibler divergence. For example, for a deep Gaussian process 300, this leads to the following expression for the KL divergence:

[0103] .

[0104] Therefore, as illustrated here, the KL divergence can be calculated based on one or more individual functions of the first and second matrices (e.g., loci and / or logarithms) without using or calculating their Kronecker product.

[0105] Now, a specific example of determining a lower bound on evidence is given in algorithmic form, which can be maximized, for example, by stochastic gradient descent (SGD) to determine the parameter set PAR:

[0106] :

[0107] - Calculate a stochastic approximation of the lower bound of the model's evidence.

[0108] - A batch of training datasets {( ,have The size of the training input TI and output TO

[0109] - Number of Monte Carlo samples

[0110] - Number of layers in the probabilistic model PM

[0111]

[0112] # Computational data fitting

[0113] for each data point do

[0114] for each repetition do

[0115]

[0116] # For example, according to Gaussian

[0117] end

[0118] end

[0119]

[0120] for each layer do

[0121]

[0122] return

[0123] As can be observed from the above instructions, in the first loop, the output of the probability model PM... Given the corresponding training input The sample is taken under certain conditions, and the corresponding training output can be determined. The likelihood is determined, and this likelihood is added to the lower bound of the evidence in a Monte Carlo manner.

[0124] Furthermore, for example, as detailed above, the Kourbach-Leibler divergence between the probability distribution approximating the outputs of the multiple Gaussian processes at multiple induced locations and the probability distribution of the outputs of the multiple Gaussian processes at multiple induced locations can be determined for each group of multiple Gaussian processes.

[0125] In order to perform training input x To sample the output of a deep Gaussian process PM, the following procedure can be used:

[0126] :

[0127]

[0128] for each layer do

[0129] distributed Target parameters

[0130] # from Meta-Gaussian distribution extraction

[0131] end

[0132] return

[0133] The probabilistic model PM can be applied to input I 470 by determining a sample SO 480 of the output of the probabilistic model PM with respect to input I through the parameter set PAR of the probabilistic model PM. In other words, sampling the output SO can be performed layer by layer and can include: determining the sample probability distribution of the output of the multiple Gaussian processes 441-443 at the layer at multiple induction locations IL for input I by marginalizing the probability distribution of the output of the multiple Gaussian processes 441-443 at the layer at multiple induction locations IL at the joint probability distribution; and sampling the output of the multiple Gaussian processes at the layer for input I according to the sample probability distribution. For example, to obtain the sample SO, the procedure drawSample given above can be used. Note that to obtain the sampled output SO, modeled output noise (e.g., noise added to the output node 330 of the depth Gaussian process 300) can be included, for example, to compare the prediction with the observed output.

[0134] Applying a probabilistic model to input I can also include determining multiple samples SO of the output of the probabilistic model PM with respect to input I. For example, based on these multiple samples, uncertainty estimates, such as variance, standard deviation, and histograms, can be derived. For example, the following procedure can be used for test data points. Identify multiple samples:

[0135] :

[0136] - Samples of the predicted distribution returned by the probability model PM

[0137] - Test data points

[0138] - Number of Monte Carlo samples

[0139] for each repetition do

[0140]

[0141] end

[0142] return

[0143] Figure 5 A block diagram of a computer-implemented method 500 for learning a parameter set of a probabilistic model is shown. The probabilistic model includes one or more layers, one of which includes multiple Gaussian processes. Method 500 may correspond to... Figure 1 The operation of system 100. However, this is not a limitation, because method 500 can also be performed using another system, apparatus or device.

[0144] Method 500 may include, in an operation entitled “Accessing the Training Dataset,” accessing 510 a training dataset comprising multiple training inputs and corresponding training outputs. Method 500 may further include, in an operation entitled “Optimizing the Parameter Set,” optimizing 520 the parameter set relative to the training dataset to determine a parameter set of the probabilistic model. The parameter set may include a set of multiple induced locations for multiple Gaussian processes. The parameter set may also include parameters approximating the probability distribution of the outputs of the multiple Gaussian processes at the multiple induced locations, wherein the probability distribution includes a multivariate normal probability distribution, the covariance matrix of which is defined by the Kronecker product of a first matrix indicating similarity among the multiple Gaussian processes and a second matrix indicating similarity among the multiple induced locations.

[0145] Figure 6 A block diagram of a computer-implemented method 600 for applying a probabilistic model to an input is shown. The probabilistic model includes one or more layers, one of which includes multiple Gaussian processes. Method 600 may correspond to... Figure 2 The operation of system 200. However, this is not a limitation, because method 600 can also be performed using another system, apparatus or device.

[0146] Method 600 may include, in an operation entitled “Accessing Inputs, Parameter Set,” accessing the inputs and parameter set of the 610 probabilistic model. The parameter set may include a set of multiple induced positions for multiple Gaussian processes. The parameter set may also include parameters approximating the probability distribution of the outputs of the multiple Gaussian processes at the multiple induced positions, wherein the probability distribution includes a multivariate normal probability distribution, the covariance matrix of which is defined by the Kronecker product of a first matrix indicating similarity among the multiple Gaussian processes and a second matrix indicating similarity among the induced positions. Method 600 may further include, in an operation entitled “Determining Output Samples,” determining samples of the outputs of the 620 probabilistic model with respect to the inputs, based on the parameter set of the probabilistic model.

[0147] To understand, generally speaking Figure 5 Method 500 and Figure 6 The operation of method 600 can be performed in any suitable order (e.g., sequentially, simultaneously, or a combination thereof), which, where applicable, is subject to a specific order that is necessary, for example, by an input / output relationship.

[0148] One or more methods can be implemented on a computer as a computer-implemented method, as dedicated hardware, or as a combination of both. For example... Figure 7 As illustrated, instructions for a computer (e.g., executable code) can be stored on a computer-readable medium 700, for example, in the form of a series of machine-readable physical marks 710 and / or as a series of elements with different electrical (e.g., magnetic) or optical properties or values. The executable code can be stored in a transient or non-transitory manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 7 An optical disk 700 is shown. Alternatively, the computer-readable medium 700 may include temporary or non-temporary data 710 representing a set of parameters as described elsewhere in this specification.

[0149] Figures 8a-8d Evaluations of probabilistic models based on various embodiments of several relevant benchmarks are shown.

[0150] The probabilistic model evaluated is a deep Gaussian process as described above, i.e., a probabilistic model comprising multiple layers of Gaussian processes, where the probability distribution approximating the output of the Gaussian process at each layer is multivariate normal, having a covariance matrix given as a Kronecker product between the first and second matrices. In some cases, the first and second matrices are general matrices, e.g., all entries of the matrices are parameters to be optimized. As discussed, this provides a model with improved coupling of the underlying Gaussian process. In other cases, the first matrix is ​​chosen as a diagonal matrix, thus providing a reduction in the number of parameters. The probabilistic model is compared with standard sparse Gaussian processes and unstructured deep GP variants (e.g., where each Gaussian process is modeled using a separate covariance matrix).

[0151] To evaluate the models, eight relevant test datasets from the UCI benchmark dataset https: / / archive.ics.uci.edu / ml / were selected. In most cases, the datasets model relatively complex nonlinear physical models. For example, the UCI “Concrete” dataset provides concrete compressive strength data as a highly nonlinear function of age and composition. The “Naval” dataset provides gas turbine maintenance status data as a function of various measurements (such as shaft torque, rotational rate, inlet and outlet temperatures and pressures) under steady-state physical conditions. The “Energy” dataset captures building heating and cooling load requirements, i.e., energy efficiency as a function of building parameters. “kin8nm” captures the forward dynamics of an 8-link full-rotation robotic arm for the purpose of predicting the distance of the end effector from the target based on joint position, torsion angle, etc. The “Power” dataset provides the net hourly electrical output of a power plant based on temperature, ambient pressure, etc. The “Protein” dataset contains data on proteins; the “Boston” dataset is about housing in Boston; and the “Wine_red” dataset allows for the determination of wine quality based on physicochemical tests.

[0152] To analyze the extrapolation behavior of the scheme, samples were classified according to the first principal component of the standardized data. The decimals with the minimum value were used for the extrapolation scenario, and the remaining 10% of the data were randomly selected for the interpolation scenario. For each dataset, the experiment was repeated with 10 independent training-interpolation splits while keeping the extrapolation sample set intact.

[0153] The number of Gaussian process layers is ,in D is the input dimension. Optimization is performed using the Adam optimizer, as described in Kingma and Ba's "Adam: A Method for Stochastic Optimization", 2014 (available at https: / / arxiv.org / abs / 1412.6980 and incorporated herein by reference). Learning rate = 0.01, batch size = The dataset has N data points. The optimizer runs for up to 20,000 iterations to minimize the evidence lower bound with respect to a set of parameters including hyperparameters, induced points, and variational parameters. Three Monte Carlo samples are used per data point for random evaluation of the lower bound. A linear mean term is used to avoid the vanishing gradient problem. The performance of the unstructured deep GP (DGP) is compared to: a deep GP (DGPkp) where the covariance matrix of each layer is a Kronecker product; and a simplified version of DGPkp (DGPkd) where the first matrix is ​​a diagonal. All deep GP models are run with 2 and 5 layers.

[0154] Figures 8a-8d The comparison of deep GP variants on UCI benchmark datasets is presented. Eight UCI benchmark datasets were investigated against the following metrics: training time 811-818, negative log-likelihood on the interpolation set 821-828, and negative log-likelihood on the extrapolation set 831-838. It was found that using the Kronecker structured covariance matrix reduces training time compared to the unstructured family 811-818, while maintaining prediction performance on 821-828 and 831-838 while keeping the number of layers constant.

[0155] When evaluating the interpolation performance of GP models of different depths, it was found that DGP... energy , kin8nm , power and protein It outperforms Sparse GP (SVGP) on the dataset and, Boston , concrete , Naval and wine The performance on the dataset is equally good. No performance differences were observed between the deep GP variants in Figures 821-828, confirming that the Kronecker approximation is accurate enough.

[0156] The runtime of the different models was measured for a single optimization step on an Intel Xeon CPU E5-1620 (3.50 GHz, 8 cores). Advantageously, a reduction in runtime was observed in Figures 811-818 from the unstructured model (DGP) to the Kronecker models (DGPkp and DGPkd) while keeping the number of layers constant. An increase in convergence speed was also observed when using the Kronecker model, attributable to the smaller number of parameters. Regarding extrapolation, for... kin8nm Consistent performance improvements were observed compared to SVGP, while performance was broadly similar on other datasets (see Figures 831-838).

[0157] Examples, embodiments, or optional features—whether or not indicated as non-limiting—should not be construed as limiting the claimed invention.

[0158] It should be noted that the embodiments mentioned above are illustrative rather than limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments without departing from the scope of the appended claims. Any reference marks placed between parentheses in the claims should not be construed as limiting the claims. The use of the verb “comprising” and its variations do not exclude the presence of elements or stages other than those described in the claims. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Expressions such as “at least one of…” when preceding a list or group of elements indicate the selection of all elements or any subset of elements from the list or group. For example, the expression “at least one of A, B, and C” should be understood to include: only A; only B; only C; both A and B; both A and C; both B and C; or all of A, B, and C. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a device claim enumerating several components, several of these components can be embodied by the same item of hardware. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used advantageously.

Claims

1. A system (100) for learning a parameter set of a probabilistic model, the probabilistic model comprising one or more layers, a layer of the one or more layers comprising a plurality of Gaussian processes, the probabilistic model predicting one or more physical output quantities based on at least one or more physical input quantities obtained from measurements obtained from various sensors to control a technical system whose performance is measured, wherein the technical system comprises a computer-controlled robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system, and wherein the sensors comprise one or more of an image or video camera, a radar, a lidar, or an ultrasonic sensor, the system comprising: - a data interface (120) for accessing a training dataset (030) comprising a plurality of training inputs and corresponding training outputs; - a processor subsystem (140) arranged to determine the parameter set of the probabilistic model (050) by optimizing the parameter set with respect to the training dataset, wherein the parameter set comprises at least: - a set of a plurality of inducing positions for the plurality of Gaussian processes; - parameters of a probability distribution approximating a variational inference of outputs of the plurality of Gaussian processes at the plurality of inducing positions, wherein the probability distribution comprises a multivariate normal probability distribution, a covariance matrix of the multivariate normal distribution being defined by a Kronecker product of a first matrix indicating a similarity between the plurality of Gaussian processes and a second matrix indicating a similarity between the plurality of inducing positions, the first matrix and the second matrix being defined by variational parameters of the variational inference approximation.

2. The system (100) of claim 1, wherein optimizing the parameter set comprises maximizing an evidence lower bound with respect to the training dataset.

3. The system (100) of claim 1 or 2, wherein optimizing the parameter set comprises sampling an output of the probabilistic model given a training input and computing a likelihood of obtaining a training output corresponding to the training input from the sampled output under a noise distribution.

4. The system (100) of claim 3, wherein sampling the output of the probabilistic model for the training input comprises: - determining a sample probability distribution of the outputs of the plurality of Gaussian processes for the training input by marginalizing out the probability distribution approximating the outputs of the plurality of Gaussian processes at the plurality of inducing positions from a joint probability distribution; - sampling the outputs of the plurality of Gaussian processes for the training input from the sample probability distribution.

5. The system (100) of any one of claims 2 to 4, wherein maximizing the evidence lower bound comprises performing a stochastic gradient descent on an objective function comprising the evidence lower bound.

6. The system (100) of any one of claims 1 to 5, wherein determining the parameter set comprises optimizing a Kullback-Leibler divergence between the probability distribution approximating the outputs of the plurality of Gaussian processes at the plurality of inducing positions and a prior probability distribution of the outputs of the plurality of Gaussian processes at the plurality of inducing positions.

7. The system (100) of any one of claims 1 to 6, wherein the first matrix is a diagonal matrix.

8. The system (100) of any one of claims 1 to 7, wherein a Gaussian process of the plurality of Gaussian processes has a squared exponential kernel.

9. The system (100) of any one of claims 1 to 8, wherein the parameter set of the probabilistic model comprises one or more kernel hyperparameters, each Gaussian process of the plurality of Gaussian processes has a kernel defined by the one or more kernel hyperparameters.

10. System (200) for applying a probabilistic model to an input, the probabilistic model comprising one or more layers, a layer of the one or more layers comprising a plurality of Gaussian processes, the probabilistic model predicting one or more physical output quantities based on at least one or more physical input quantities obtained from measurements obtained from various sensors to control a technical system whose performance is measured, wherein the technical system comprises a computer-controlled robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system, and wherein the sensors comprise one or more of an image or video camera, a radar, a lidar, or an ultrasonic sensor, the system comprising: - a data interface (220) for accessing an input (060) and a parameter set (050) of the probabilistic model, wherein the parameter set comprises at least: - a set of multiple inducing positions for the plurality of Gaussian processes; - parameters of a probability distribution approximating a variational inference of outputs of the plurality of Gaussian processes at the multiple inducing positions, wherein the probability distribution comprises a multivariate normal probability distribution, a covariance matrix of the multivariate normal distribution being defined by a Kronecker product of a first matrix indicating similarities between the plurality of Gaussian processes and a second matrix indicating similarities between the inducing positions, the first matrix and the second matrix being defined by variational parameters of the variational inference approximation; - a processor subsystem (240) arranged to determine a sample of an output of the probabilistic model with respect to the input (060) based on the parameter set (050) of the probabilistic model.

11. The system (200) of claim 10, wherein the processor is configured to determine a plurality of samples of the output of the probabilistic model with respect to the input and to determine therefrom an uncertainty estimate.

12. Method (500) of learning a parameter set of a probabilistic model, the probabilistic model comprising one or more layers, a layer of the one or more layers comprising a plurality of Gaussian processes, the probabilistic model predicting one or more physical output quantities based on at least one or more physical input quantities obtained from measurements obtained from various sensors to control a technical system whose performance is measured, wherein the technical system comprises a computer-controlled robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system, and wherein the sensors comprise one or more of an image or video camera, a radar, a lidar, or an ultrasonic sensor, the method comprising: - accessing (510) a training data set comprising a plurality of training inputs and corresponding training outputs; - determining the parameter set of the probabilistic model by optimizing (520) the parameter set with respect to the training data set, wherein the parameter set comprises at least: - a set of multiple inducing positions for the plurality of Gaussian processes; - parameters of a probability distribution that is a variational inference approximation of outputs of the plurality of Gaussian processes at the plurality of induced locations, wherein the probability distribution comprises a multivariate normal probability distribution, a covariance matrix of the multivariate normal distribution being defined by a Kronecker product of a first matrix indicative of similarities between the plurality of Gaussian processes and a second matrix indicative of similarities between the plurality of induced locations, the first matrix and the second matrix being defined by variational parameters of the variational inference approximation.

13. A method (600) of applying a probabilistic model to an input, the probabilistic model comprising one or more layers, one of the one or more layers comprising a plurality of Gaussian processes, the probabilistic model predicting one or more physical output quantities based on at least one or more physical input quantities obtained from measurements obtained from various sensors to control a technical system of which a performance is measured, wherein the technical system comprises a computer-controlled robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system, and wherein the sensors comprise one or more of an image or video camera, a radar, a lidar, or an ultrasonic sensor, the method comprising: - accessing (610) an input and a parameter set of the probabilistic model, wherein the parameter set comprises at least: - a set of a plurality of induced locations for the plurality of Gaussian processes; - parameters of a probability distribution that is a variational inference approximation of outputs of the plurality of Gaussian processes at the plurality of induced locations, wherein the probability distribution comprises a multivariate normal probability distribution, a covariance matrix of the multivariate normal distribution being defined by a Kronecker product of a first matrix indicative of similarities between the plurality of Gaussian processes and a second matrix indicative of similarities between the induced locations, the first matrix and the second matrix being defined by variational parameters of the variational inference approximation; - determining (620) a sample of an output of the probabilistic model with respect to the input based on the parameter set of the probabilistic model.

14. A computer-readable medium (700) comprising transitory or non-transitory data (710) representing instructions arranged to cause a processor system to perform a computer-implemented method according to claim 12 or 13.

15. A computer-readable medium (700) comprising transitory or non-transitory data (710) representing a parameter set of a probabilistic model, the probabilistic model comprising one or more layers, one of the one or more layers comprising a plurality of Gaussian processes, the probabilistic model predicting one or more physical output quantities based on at least one or more physical input quantities obtained from measurements obtained from various sensors to control a technical system of which a performance is measured, wherein the technical system comprises a computer-controlled robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system, and wherein the sensors comprise one or more of an image or video camera, a radar, a lidar, or an ultrasonic sensor, the parameter set comprising at least: - a set of a plurality of induced locations for the plurality of Gaussian processes; - performing variational inference approximation of parameters of a probability distribution over outputs of a plurality of Gaussian processes at a plurality of inducing locations, wherein the probability distribution comprises a multivariate normal probability distribution, a covariance matrix of the multivariate normal distribution being defined by a Kronecker product of a first matrix indicative of similarities between the plurality of Gaussian processes and a second matrix indicative of similarities between the inducing locations, the first matrix and the second matrix being defined by variational parameters of the variational inference approximation.

Images