Method and apparatus for processing sensor data

Abstract

According to various embodiments, there are provided methods and apparatuses for processing sensor data, the method comprising: receiving input sensor data; determining a plurality of end states from the input sensor data as an initial state, including determining a state sequence for each end state, wherein determining a state sequence comprises, for each state of a sequence starting with the initial state until an end state, a first Bayesian neural network determining a drift term sample in response to inputting the respective state; a second Bayesian neural network determining a diffusion term sample in response to inputting the respective state; and determining a subsequent state by sampling a stochastic differential equation comprising the drift term sample as a drift term and the diffusion term sample as a diffusion term. The method further comprises determining an end state probability distribution from the determined plurality of end states, and determining a processing result for the input sensor data from the end state probability distribution.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a method and a device for processing sensor data. BACKGROUND

[0002] The results of a regression analysis of sensor data can be applied to various control tasks. For example, in an autonomous driving scenario, a vehicle can perform a regression analysis of sensor data indicative of a road curvature to derive a maximum speed. However, in many applications, it is not only relevant what the result is (e.g. the maximum speed in the above example), but also how certain the result is. For example, in an autonomous driving scenario, a vehicle controller should take into account whether a prediction of the maximum possible maximum speed has a sufficient certainty before controlling the vehicle accordingly.

[0003] The publication "Neural Ordinary Differential Equations" by Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud (NeurIPS, 2018) introduces a neural network governing the dynamics of an ordinary differential equation (ODE) as a general building block in learning systems. An input pattern is set as the initial value of the ODE. However, this is a fully deterministic dynamics system, so it cannot express uncertainty. SUMMARY

[0004] In view of the above, a flexible machine learning method providing uncertainty information for the output is desirable.

[0005] The method and the device having the features of independent claim 1 (corresponding to example 1 below) and independent claim 7 (corresponding to example 7 below) allow for achieving an improved robustness compared to deterministic methods by modeling the flow dynamics as a stochastic differential equation (SDE) and quantifying the prediction uncertainty. In particular, the robustness is improved by distributing a Bayesian neural network (BNN) over the drift and diffusion terms of the SDE. By using a BNN in this way, a second source of randomness from the BNN weights (in addition to the Wiener process for the diffusion) is introduced, which improves the robustness and the quality of the prediction uncertainty distribution.

[0006] In addition, the method and the device according to the independent claims do not require a manual dropout rate adjustment compared to dropout-based methods and provide a richer solution family than fixed-rate dropout.

[0007] In the following, various examples are given.

[0008] Example 1 is a method for processing sensor data, the method comprising receiving input sensor data; determining a plurality of end states from the input sensor data as an initial state, including determining a state sequence for each end state, wherein determining a state sequence comprises, for each state of a sequence starting with the initial state until an end state, a first Bayesian neural network determining a drift term sample in response to inputting the respective state; a second Bayesian neural network determining a diffusion term sample in response to inputting the respective state; and determining a subsequent state by sampling a stochastic differential equation, the stochastic differential equation comprising the drift term sample as a drift term and the diffusion term sample as a diffusion term; determining an end state probability distribution from the determined plurality of end states; and determining a processing result for the input sensor data from the end state probability distribution.

[0009] Example 2 is the method according to example 1, further comprising training the first Bayesian neural network and the second Bayesian neural network using stochastic gradient Langevin dynamics.

[0010] SGLD allows to infer model parameters, circumventing drawbacks of variational inference such as limited expressivity of the approximating distribution.

[0011] Example 3 is the method according to example 1 or 2, wherein the processing result comprises a control value and uncertainty information about the control value.

[0012] The uncertainty information allows to identify erroneous predictions of the model (or at least predictions for which the model is not confident) and thus to avoid erroneous control decisions.

[0013] Example 4 is the method according to example 3, wherein determining the end state probability distribution comprises estimating a mean vector and a covariance matrix of the end states, and wherein determining the processing result from the end state probability distribution comprises determining a predicted mean from the estimated mean vector of the end states and a predicted variance from the estimated covariance matrix of the end states.

[0014] The end state of a vector-valued can thus be reduced to a one-dimensional value (including the uncertainty information in terms of variance), which can for example be used for actuator control.

[0015] Example 5 is the method according to example 4, wherein determining the processing result comprises processing the estimated mean vector and the estimated covariance matrix by a linear layer, the linear layer performing an affine mapping of the estimated mean vector to a one-dimensional predicted mean and a linear mapping of the estimated covariance matrix to a one-dimensional predicted variance.

[0016] The linear derivation of the processing result allows a proper propagation of the uncertainty information from the end state probability distribution to the processing result.

[0017] Example 6 is the method according to any one of examples 1 to 5, comprising using the processing result to control an actuator.

[0018] Controlling an actuator based on the method of the first example allows to ensure a safe control of e.g. a vehicle.

[0019] Example 7 is a neural network device adapted to perform the method according to any one of examples 1 to 6.

[0020] Example 8 is a software or hardware agent, in particular a robot, comprising a sensor adapted to provide sensor data and a neural network device according to example 7, wherein the neural network device is configured to perform a regression or classification of the sensor data.

[0021] Example 9 is the software or hardware agent according to example 8, comprising an actuator and a controller configured to use an output from the neural network device to control the at least one actuator.

[0022] Example 10 is a computer program comprising computer instructions which, when executed by a computer, cause the computer to perform the method according to any one of examples 1 to 6.

[0023] Example 11 is a computer readable medium comprising computer instructions which, when executed by a computer, cause the computer to perform the method according to any one of examples 1 to 6. BRIEF DESCRIPTION OF DRAWINGS

[0024] In the drawings, like reference numerals refer to like parts throughout the various views Figure One of the present disclosure. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure. In the following description, various aspects of the disclosure will be described in reference to the following drawings, in which:

[0025] Figure 1 An example for a regression in an autonomous driving scenario is shown.

[0026] Figure 2 A diagram of a machine learning model according to an embodiment is shown.

[0027] Figure 3 A flowchart illustrating a method for processing sensor data according to an embodiment is shown. DETAILED DESCRIPTION

[0028] The following detailed description references the drawings, which illustrate specific details of particular embodiments of the disclosure by way of specific illustrative examples. Other embodiments can be utilized, and structural, logical, and electrical changes can be made without departing from the scope of the present disclosure. The various aspects of the disclosure are not necessarily mutually exclusive, as some aspects of the disclosure can be combined with one or more other aspects of the disclosure to form new aspects.

[0029] In the following, various examples will be described in more detail.

[0030] Figure 1 An example is shown for regression in an autonomous driving scenario.

[0031] In Figure 1 In the example, a vehicle 101, such as a car, truck or motorcycle, is provided with a vehicle controller 102.

[0032] The vehicle controller 102 comprises a data processing component, such as a processor, e.g. a CPU (Central Processing Unit), 103, and a memory 104 for storing control software according to which the vehicle controller 102 operates and data on which the processor 103 operates.

[0033] In the example, the stored control software comprises instructions which, when executed by the processor 103, cause the processor to implement a regression algorithm 105.

[0034] The data stored in the memory 104 can comprise input sensor data from one or more sensors 107. For example, the one or more sensors 107 can comprise sensors measuring speed of the vehicle 101 and sensor data indicative of road curvature, which can for example be derived from image sensor data by object detection processing for determining road direction, road conditions, etc. Thus, the sensor data can for example be multi-dimensional (curvature, road conditions,...). The regression result can for example be one-dimensional.

[0035] The vehicle controller 102 processes the sensor data and determines a regression result, such as a maximum speed, and can use the regression result to control the vehicle. For example, if the regression result indicates a maximum speed which is higher than a measured current speed of the vehicle 101, it can actuate an interrupt 108.

[0036] The regression algorithm 105 can comprise a machine learning model 106. The machine learning model 106 can be trained using training data to make predictions, such as maximum speed. Due to safety concerns related to the control task, the machine learning model 106 can be chosen which not only outputs a regression result, but also outputs an indication of its certainty about the regression result. When controlling the vehicle 101, the controller 102 can take this certainty into account, e.g. in case the certainty of a prediction is low, e.g. below a predetermined threshold, it brakes even if it is below the predicted maximum speed.

[0037] A widely used machine learning model is a deep neural network. A deep neural network is trained to implement a function that non-linearly transforms input data (in other words, input patterns) into output data (output patterns). If the neural network is a residual neural network, its processing pipeline can be seen as an ODE (ordinary differential equation) system discretized across even time intervals. Re-formulating the model in terms of continuous-time ODEs is known as neural ODE.

[0038] According to various embodiments, a universal Bayesian neural model is provided (which can be used as the machine learning model 106, for example) which comprises solving SDEs (stochastic differential equations) as an intermediate step to model the flow of activation graphs. The drift and diffusion functions of the SDEs are implemented as a Bayesian neural net (BNN).

[0039] According to the neural-ODE approach, the processing of a neural network is formulated as:

[0040] ,

[0041] where reflects the parameters of the neural network, and is the output of layer . This can be interpreted as an explicit Euler scheme for solving an ODE with step size 1.

[0042] With this interpretation, the equation above can be reformulated as:

[0043] .

[0044] Thus, ODE calculus can be used to propagate through a neural network. To make this equation stochastic, consider stochastic ordinary differential equations. In general form, they are given as:

[0045] .

[0046] The equation is governed by a drift and a diffusion The drift models the deterministic part, the diffusion Model the random component. For This yields the standard ODE. Solving the above equations requires considering Brownian motion. Integral, the Brownian motion This reflects the stochastic part of the differential equation. A common and easy approximation method for this differential equation is the Euler-Maruyama scheme:

[0047] ,

[0048] It is a Gaussian random variable with the following properties:

[0049] .

[0050] When variables It is a vector This approximation also holds true in this case. In this situation, the diffusion term is a matrix-valued function of the input and time. And the corresponding Modeled as independent of P Wiener process ,in As P 3D identity matrix.

[0051] As stated above, according to various embodiments, and Each is provided by a corresponding Bayesian neural network (BNN), in which computation... The weights of the BNN are determined by Mark, and calculate The weights of the BNN are determined by Indication. Weights can be shared at least partially among BNNs, i.e. .

[0052] The resulting probabilistic machine learning model can be described by the following formula.

[0053] .

[0054] The first line is the prior information about the SDE parameters (in this case, the weights of the BNN), the second line is the solution to the SDE, and the last line is the probability of fitting the output space of the machine learning model. T It is the duration of the flow corresponding to the model capacity.

[0055] Figure 2 A diagram of a machine learning model is shown.

[0056] The input is a vector x.

[0057] For an (input observation) vector as initial condition x , representing a continuous-time activation map h(t) The realization of the stochastic process 201 is determined as a solution of an SDE. All t of h(t) where e.g. can be seen as an input pattern x at each time instant t The latent representation. The part of the machine learning model that makes this determination is called a differential Bayesian neural network (DBNN). It comprises BNNs 202, 203 that provide the mean term and the diffusion term of the SDE, respectively (each taking h(t) and t as input). The DBNN outputs an output value h(T) which can be a vector of the same dimension as the input vector x .

[0058] Depending on the application, an additional (e.g. linear) layer 204 computes the output y of the model, e.g. a regression result for an input sensor data vector x . This additional layer 204 can in particular reduce the dimensionality of h(T) (which can be seen as an end state) to a desired output dimension, e.g. generating a real number y from a vector h(T) .

[0059] The probability distribution of the stochastic process is given by

[0060]

[0061] where is a Brownian motion corresponding to the Wiener process . It should be noted that the second integral on the right-hand side of the equation is an Itô integral, unlike the first integral. The associated SDE is

[0062]

[0063] where is a drift term governing the dynamics of the flow, and is a diffusion term that makes the motion jitter at each instant. The probability does not have a closed-form expression that generalizes across all neural network architectures. However, it is possible to draw approximate samples from it by discretization rules such as Euler-Maruyama.

[0064] According to one embodiment, as a strain method, the stochastic process is excluded from the likelihood by Monte Carlo integration according to

[0065]

[0066] in It is the first m The realization of the Euler-Maruyama draw at time T. Given that the stochastic process has already been integrated, it can be achieved by considering... The approximate posterior inference problem is used to train the model. For stochastic processes... h The sample-driven solutions are naturally integrated into the Markov Chain Monte Carlo (MCMC) scheme. According to one embodiment, stochastic gradient Langevin dynamics (SGLD) with block decay structure is used to benefit from the gradient descent algorithm as a subroutine (which is necessary for effectively training the neural network).

[0067] The following describes a training algorithm for a model, specifically for supervised learning to determine the model from training data (including multiple mini-batches). θ 1 and θ 2 The algorithm.

[0068]

[0069] It should be noted that backpropagation can be used to determine the gradient. It should be further noted that, θ 1 probability distribution and θ 2 The probability distribution can be determined by storing the values ​​of the latest iteration (e.g., for the last 100i) to reach the trained BNN 202, 203.

[0070] For regression, the additional linear layer 204 was placed h(T) Above, to match the output dimension. Due to the distribution Properties can be found in the mean Covariance (Cholesky decompose) The two moments are estimated. They can be determined and then propagated through a linear layer 204. Therefore, the predicted mean is modeled as... And the prediction variance is modeled as Assuming unrelated activation graph dimensions, It is possible to design it as a diagonal matrix.

[0071] In addition, it can be done by... The Joleski decomposition assigns the DBNN output to make Parameterization, or it can be taken in the form of any other structure. When P < D, it is possible to severely reduce the number of learnable parameters of the high-dimensional input.

[0072] In summary, according to various embodiments, there is provided a method as illustrated in Figure 3

[0073] Figure 3 A flowchart 300 illustrating a method for processing sensor data according to an embodiment is shown.

[0074] In 301, input sensor data is received.

[0075] In 302, a plurality of end states is determined starting from the input sensor data as an initial state.

[0076] This comprises determining for each end state a state sequence, wherein determining the state sequence comprises determining for each state of a sequence starting with the initial state until the end state,

[0077] a first Bayesian neural network determines a drift term sample in response to inputting the respective state;

[0078] a second Bayesian neural network determines a diffusion term sample in response to inputting the respective state; and

[0079] a subsequent state is determined by sampling a stochastic differential equation, the stochastic differential equation comprising the drift term sample as a drift term and the diffusion term sample as a diffusion term.

[0080] In 303, an end state probability distribution is determined from the determined plurality of end states.

[0081] In 304, a processing result of the input sensor data is determined from the end state probability distribution.

[0082] According to various embodiments, in other words, at each step of solving the stochastic differential equation, a BNN is used to provide the drift term and the diffusion term. In addition to the uncertainty information provided by solving the stochastic differential equation (by sampling the Brownian motion), the uncertainty information provided by the BNN (by sampling the BNN weights) also provides information for the processing result, e.g. a regression result, e.g. for controlling a device depending on the sensor data.

[0083] Figure 3 The method can be used as a general building block in all learning systems that map input patterns to output patterns. It can act as an intermediate processing step that provides a rich family of mappings that can then be tuned to a specific dataset. As long as feedforward neural networks can be used, the method can be used. Figure 3 ​The method of the application is particularly useful in safety-critical applications, where a prediction needs to be validated before a downstream action is taken depending on the computer system's prediction, or where the uncertainty of the prediction needs to be taken into account.

[0084] In particular, Figure 3 The method of the application can be applied to all supervised learning settings where a likelihood distribution can be expressed for the output (e.g. a normal distribution for continuous outputs, a multinomial distribution for discrete outputs). Furthermore, it can be applied to any generative method where the latent representation has the same dimensionality as the observation. It can further be applied in hypernets using synthetic BNN weight distributions as approximate distributions in inference problems such as variational inference. Examples of applications are image segmentation and reinforcement learning.

[0085] Figure 3 The method of the application can be executed by one or more computers comprising one or more data processing units. The term "data processing unit" can be understood as any kind of entity allowing data or signal processing. For example, data or signals can be handled in accordance with at least one (i.e. one or more than one) specific function performed by the data processing unit. The data processing unit can comprise or be formed by an analog circuit, a digital circuit, a complex signal circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA) integrated circuit, or any combination thereof. Any other means of realizing a respective function, which will be described in more detail below, can also be understood as a data processing unit or a logic circuit. It will be understood that one or more method steps described in detail herein can be performed by the data processing unit by one or more specific functions performed by the data processing unit (e.g. to implement).

[0086] The first and the second Bayesian neural network can be trained by comparing, for each of the plurality of training data units, a processing result of the training data unit for the input sensor training data with a reference value of the training data unit.

[0087] Generally, Figure 3 The method of the application can be used to generate control data, e.g. data for controlling a robot, from input sensor data. The term "robot" can be understood to refer to any physical system (with mechanical components whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.

[0088] Neural networks can be used for regression or classification on data. The term classification is understood to include semantic segmentation, e.g. of images (which can be seen as pixel-wise classification). The term classification is also understood to include detection, e.g. of objects (whether or not the object is present, it can be seen as a classification). Regression includes in particular time series modeling.

[0089] While specific embodiments have been illustrated and described herein, it will be appreciated that various alternative and / or equivalent implementations can be substituted for the specific embodiments shown and described without departing from the scope of the present disclosure. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that the application be limited only by the claims and the equivalents thereof.

Claims

1. A method for processing sensor data, the method comprising: receiving input sensor data; determining a plurality of end states from the input sensor data as an initial state, including determining a state sequence for each end state, wherein determining a state sequence includes determining for each state of a sequence starting with the initial state until the end state, a first Bayesian neural network determining a drift term sample in response to inputting a respective state; a second Bayesian neural network determining a diffusion term sample in response to inputting a respective state, wherein weights of the first Bayesian neural network and weights of the second Bayesian neural network are at least partially shared between the first Bayesian neural network and the second Bayesian neural network; and determining a subsequent state by sampling a stochastic differential equation, the stochastic differential equation including the drift term sample as a drift term and the diffusion term sample as a diffusion term; determining an end state probability distribution from the determined plurality of end states; and determining a processing result for the input sensor data from the end state probability distribution.

2. The method according to claim 1, further comprising: training the first Bayesian neural network and the second Bayesian neural network using stochastic gradient Langevin dynamics.

3. The method according to claim 1 or 2, wherein the processing result includes a control value and uncertainty information about the control value.

4. The method according to claim 3, wherein determining the end state probability distribution includes estimating a mean vector and a covariance matrix of the end state, and wherein determining the processing result from the end state probability distribution includes determining a predicted mean from the estimated mean vector of the end state and a predicted variance from the estimated covariance matrix of the end state.

5. The method according to claim 4, wherein determining the processing result includes processing the estimated mean vector and the estimated covariance matrix by a linear layer performing an affine mapping of the estimated mean vector to a one-dimensional predicted mean and a linear mapping of the estimated covariance matrix to a one-dimensional predicted variance.

6. The method according to claim 1 or 2, including controlling an actuator using the processing result.

7. A neural network device adapted to perform the method according to any one of claims 1 to 6.

8. A robot, comprising a sensor adapted to provide sensor data; and a neural network device according to claim 7, wherein the neural network device is configured to perform a regression or a classification of the sensor data.

9. The robot according to claim 8, including an actuator and a controller configured to control at least one actuator using an output from the neural network device.

10. A computer program comprising computer instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.

11. A computer readable medium comprising computer instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.

Images