Control of physical systems based on inferred states

Abstract

Control of a physical system based on an inferred state is provided. A system (100) and computer-implemented method are provided for enabling control of a physical system (60) based on a state of the physical system inferred from sensor data (122). The system and method may iteratively infer the state by iteratively performing the following: obtaining an initial inference of the state using a mathematical model representing a priori knowledge-based modeling of the state; and by applying a learned model to the initial inference of the state and sensor measurements, wherein the learned model has been learned to minimize the error between the initial inference provided by the mathematical model and the ground truth, and providing a correction value as an output for correcting the initial inference of the state of the mathematical model. Output data (142) may be provided to an output device (40) to enable control of the physical system based on the inferred state.

Description

Technical Field

[0001] The present invention relates to a system and a computer-implemented method for enabling control of a physical system based on a state of the physical system inferred from sensor data. The present invention further relates to a computer-readable medium comprising data representing instructions, the instructions being arranged to cause a processor system to perform the computer-implemented method. Background Art

[0002] It is known to control a physical system based on the state of the physical system determined from sensor data. For example, the movement of an autonomous agent such as a robot, car or drone may be controlled based on the agent's current position, where the current position is determined from position data obtained from a Global Positioning System (GPS) sensor or similar type of geolocation sensor. However, such position data does not represent the "true" position of the agent, but rather sensor-based measurements that may deviate from the true position due to uncertainty in the measurements, which is also simply referred to as "noise" and may be due to various reasons, such as, in this example, due to satellite signal blockage, multipath interference, radio interference, etc.

[0003] However, it is desirable to be able to determine the true position of the agent as accurately as possible based on such potentially noisy sensor data. Similar examples where it is desirable to determine the state of a physical system based on potentially noisy sensor data include: a heating system for a building (as an example of a physical system), where it may be desirable to use sensor data obtained from a temperature sensor to determine the temperature (as an example of a state); or an electric motor (as another example of a physical system), where it may be desirable to determine the position of the rotor (as another example of a state) based on sensor data obtained from a position feedback sensor.

[0004] It will be appreciated that sensor measurements may be associated with state, but typically represent measurements of observable quantities, whereas the state itself may be an unobservable quantity, e.g., for fundamental or practical reasons. For example, temperature is typically measured indirectly, e.g., using a temperature sensor that measures changes in resistance, where these changes in resistance are correlated to changes in temperature. Other examples of sensor measurements that are only indirectly related to a particular state of a physical system include, but are not limited to, the following: the orientation of a device (as an example of state) may be determined from an accelerometer that measures so-called "proper" acceleration, and the occupancy of a room (as another example of state) may be determined from infrared measurements obtained in the room.

[0005] It is known to filter sensor data to reduce noise in the sensor data, thereby allowing a more accurate determination of state from such sensor data. For example, a Kalman filter can be used to recursively estimate the state of a physical system based on a time series of sensor measurements. Such a Kalman filter can incorporate prior knowledge about the relationship between the sensor measurements and the state of the physical system. For example, a mathematical model can be used that represents a prior knowledge-based modeling of the state as a function of the sensor measurements and the previously inferred state. Such a mathematical model can, for example, incorporate the physical laws of motion to estimate the current position of the robot based on the robot's previous position and geolocation measurements obtained from a GPS sensor in the robot. Summary of the invention

[0006] It may be desirable to enable control of a physical system based on a state of the physical system inferred from sensor data, where the inference of the state is improved upon Kalman filtering and similar techniques.

[0007] According to a first aspect of the invention, there is provided a system for enabling control of a physical system as defined in claim 1. According to a further aspect of the invention, there is provided a computer-implemented method for enabling control of a physical system as defined in claim 13. According to a further aspect of the invention, there is provided a computer-readable medium as defined in claim 14 and a computer-readable medium as defined in claim 15.

[0008] The above measures provide for iterative inference of the state of a physical system based on sensor data representing sensor measurements associated with the state of the physical system. The physical system may be, for example, a physical entity such as a vehicle, a robot, etc., or a connected or distributed system of physical entities such as a lighting system, or any other type of physical system such as a building. The state may be a quantity that varies over time, such as temperature, velocity, acceleration, position, geolocation, occupancy, etc., but may also be a set of such quantities, such as a vector of velocity and acceleration.

[0009] The state may be inferred from sensor data that may be obtained from one or more sensors. The sensor(s) may be part of a physical system so that sensor measurements associated with the state of the physical system can be obtained, or may be arranged separate from the physical system if the sensor measurements may be obtained remotely. The sensor measurements are associated with quantities that allow the state to be inferred from the sensor measurements. As such, there is a correlation between the measured quantity(s) and the quantity(s) representing the state to be inferred. Typically, the quantity representing the state to be inferred cannot be directly measured, e.g. for fundamental or practical reasons.

[0010] The above measures involve iteratively inferring the state of a physical system. For this purpose, in an iteration of inference, sensor measurements are obtained, after which an initial inference of the state is obtained using a mathematical model representing a prior knowledge-based modeling of the state. More specifically, the sensor measurements and the previously inferred state are used as inputs to the mathematical model, which then produces an initial inference of the state as output. The mathematical model can encode prior knowledge about the relationship between the sensor measurement and the state to be inferred, wherein the model takes the previously inferred state into account. For example, the mathematical model can represent the physical modeling of the relationship between the sensor measurement and the state to be inferred. In a specific example, if the state is the temperature to be measured and the measured quantity is the resistance of the material, the mathematical model can express the temperature as a function of the measured resistance and the previously inferred temperature. In generating the mathematical model, domain knowledge, such as the temperature coefficient of resistance (TCR) of the material, can be used. In general, the mathematical model can be constructed, for example, by a system designer at least partially based on a manual specification and expressed as one or a set of equations. Such a type of mathematical model itself can be known from the fields of statistics and control systems.

[0011] The above measures further involve providing a learned model and applying the learned model to the initial inference of state and sensor measurements. The learned model, which may represent any suitable learned model (such as a neural network), has been learned to minimize the error between the initial inference provided by the mathematical model and the ground truth. A correction value is provided as an output for correcting the initial inference of the state of the mathematical model. The state of the physical system associated with the sensor measurements is then obtained by combining the initial inference of the state provided by the mathematical model and the correction value provided by the learned model, for example by applying the correction value to the initial inference of the state by means of simple addition.

[0012] The above measures have the following effect: the mathematical model is used to obtain an initial estimate of the state of the physical system, which is then corrected by a learned model that has been specifically learned to correct the initial inferences of the mathematical model based on basic facts. In fact, prior knowledge-based modeling by the mathematical model can provide a rough estimate of the state of the physical system, while the learned model can provide a refinement of the rough estimate. That is, learning by relying on basic facts may have enabled the learned model to identify unknown relationships or correlations between sensor measurements and the state to be inferred, which have not yet been represented in the prior knowledge-based modeling by the mathematical model.

[0013] Surprisingly, it has been found that such a hybrid approach using a learned model and a prior knowledge-based ("non-learned") mathematical model improves the accuracy of state inference - not only compared to using prior knowledge-based modeling alone, but also compared to using a learned model that has been learned to directly infer the state of the physical system based on sensor measurements. That is, such a "direct" learned model may have difficulty accurately modeling the dynamics of the physical system, for example due to insufficient complexity of the learned model and / or due to insufficient availability of training data. In contrast, the learned model used in the above measures may be less complex and / or require less training data, because the (one or more) relationships between sensor measurements and the state may have been modeled to an approximate degree by the mathematical model. As such, training may be limited to typically small deviations from the ground truth that must be learned in the initial inference.

[0014] The state of the physical system has been inferred based on sensor measurements, and output data can be provided to an output device used in the control of the physical system to enable control of the physical system based on the inferred state. For example, the output device can be an actuator that is part of the physical system or located near the physical system. The output data can be used to control the actuator and thereby control the physical system. In other examples, the output device can be a presentation device that is configured to generate a sensory perceptible output signal based on the inferred state. For example, the presentation device can be a display configured to display the output signal. Other types of sensory perceptible output signals include, but are not limited to, light signals, auditory signals, tactile signals, etc. The corresponding presentation device itself is known. Based on the sensory perceptible output signal, the operator can then control the physical system. For example, the sensory perceptible output signal can indicate a fault in a component of the physical system, and the fault can prompt the operator to control the physical system by stopping or pausing the operation of the physical system. In general, the sensory perceptible output signal can represent an inferred state, or can represent a result derived from the inferred state, such as fault diagnosis.

[0015] Optionally, the processor subsystem is configured to iteratively infer the state of the physical system by using a time series of sensor measurements and a previous inferred state, as also defined by claim 2. In practice, instead of using only previous sensor measurements and previous inferred states, the corresponding time series can be used as input to the mathematical model and the learned model. Each time series can be restricted in time range and can be represented by a sliding window, which can be implemented as a circular buffer, for example. The use of such time series can provide a more accurate state inference, because the dynamics of the physical system can be better approximated by the mathematical model and / or such an approximation can be more accurately refined by the learned model.

[0016] Optionally, the learned model is a recurrent neural network (RNN), and the processor subsystem is configured to maintain and pass hidden states of the recurrent neural network between iterations of iterative inference of the state of the physical system. Such a recurrent neural network is well suited as a learned model when iteratively inferring the state of the physical system because such types of neural networks exhibit temporal dynamic behavior and can therefore be used to learn refinements of the state approximation of a dynamic physical system. For example, the learned model can be a graph neural network (GNN) that includes a gated recurrent unit (GRU) to establish recurrence in the graph neural network.

[0017] Optionally, the mathematical model includes a transition model portion that models the conditional probability of the state to be inferred given a previous inferred state, and a measurement model portion that models the conditional probability of sensor measurements of the state to be inferred. The mathematical model can therefore be a probability-based model that models the state as at least two conditional probabilities: a first "transition" probability that represents the conditional probability of the previous inferred state transitioning to the current inferred state; and a second "measurement" probability that represents the conditional probability that the sensor measurement indicates the current inferred state. This type of prior knowledge-based modeling is well suited to being handled by mathematical estimation techniques such as Kalman filtering, for example, when the probability distributions of the mathematical model are assumed to be linear and Gaussian, respectively.

[0018] It will be appreciated by those skilled in the art that two or more of the above-mentioned embodiments, implementations, and / or alternative aspects of the invention may be combined in any way deemed useful.

[0019] On the basis of this description, those skilled in the art may implement modifications and variations of the computer-implemented method or any computer-readable medium corresponding to the described modifications and variations of the system, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] These and other aspects of the invention will be apparent from and further elucidated with reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, in which:

[0021] Figure 1 A system for enabling control of a physical system is shown, wherein the system is configured to iteratively infer a state of the physical system based on sensor data obtained from a sensor and provide output data to an output device (such as an actuator) associated with the control of the physical system;

[0022] Figure 2 The diagram shows a hidden Markov process;

[0023] Figure 3 illustrates iterations of iterative inference in an embodiment, wherein inference is modeled as a message passing scheme;

[0024] Figure 4 A method for enabling control of a physical system is shown; and

[0025] Figure 5 A computer readable medium including data is shown.

[0026] It should be noted that the figures are purely diagrammatic and are not drawn to scale.In the figures, elements corresponding to elements already described may have the same reference numerals.

[0027] Reference Numbers List

[0028] The following list of reference numerals is provided to facilitate the interpretation of the drawings and should not be construed as limiting the claims.

[0029] 20 Sensors

[0030] 40 Output Devices

[0031] 60 Physics System

[0032] 100 Systems for enabling control of physical systems

[0033] 120 Input interface

[0034] 122 Sensor data

[0035] 140 Output interface

[0036] 142 Output data

[0037] 160 Processor Subsystem

[0038] 180 Data storage interface

[0039] 190 Data storage device

[0040] 192 Data Representation of Mathematical Models

[0041] 194 Data Representation of Learned Models

[0042] 200 Methods for enabling control of physical systems

[0043] 210 Accessing sensor data

[0044] 220, 222 Iteration of iterative inference

[0045] 230 Get sensor measurement

[0046] 240 Obtaining Initial Inferences from Mathematical Models

[0047] 250 Get correction value from learned model

[0048] 260 Combining initial inferences with corrections

[0049] 270 Provide output data to output device

[0050] 300 Computer readable medium

[0051] 310 Non-transitory data. DETAILED DESCRIPTION

[0052] The following relates to a system and computer-implemented method for enabling control of a physical system based on a state of the physical system inferred from sensor data. Specific examples of such physical systems, states to be inferred, and types of sensor data have been indicated in the overview section. Various aspects of implementation of the system and computer-implemented method, as well as the iterative inference itself, are described in detail below.

[0053] Figure 1 A system 100 is shown for enabling control of a physical system based on a state of the physical system inferred from sensor data. Figure 1 The physical system is schematically shown as a dashed outline 60, but in general, such a physical system may be, for example, a physical entity such as a vehicle, a robot, etc.; or a connected or distributed system of physical entities, such as a lighting system; or any other type of physical system, such as a building. The physical system 60 is shown to include a sensor 20 that can measure one or more quantities related to the state of the physical system to be inferred. Figure 1As shown in , the sensor 20 can be part of the physical system. In other examples, the sensor 20 can be arranged remotely from the physical system 60, for example if the (one or more) quantities can be measured remotely. For example, a camera-based sensor can be arranged external to the robot, but can still measure quantities associated with the robot, such as the position and orientation of the robot within the workspace.

[0054] The system 100 is shown to include an input interface 120, which is shown to access sensor data 122 from the sensor 20. In practice, the input interface 120 may represent a sensor interface. Alternatively, the input interface 120 may access sensor data from elsewhere, such as from a data storage device or a network location. Thus, the input interface 120 may have any suitable form, including but not limited to: a low-level communication interface, such as based on I2C or SPI data communications; but also a data storage interface, such as a memory interface or a permanent storage interface; or a personal, local or wide area network interface, such as a Bluetooth, ZigBee or Wi-Fi interface, or an Ethernet or fiber optic interface.

[0055] The system 100 is further shown to include an output interface 140 to provide output data 142 to an output device 40, which may be, for example, an actuator 40 as part of the physical system 60. For example, the actuator may be an electrical, hydraulic, pneumatic, thermal, magnetic, and / or mechanical actuator. Specific, yet non-limiting, examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servomechanisms, solenoids, stepper motors, and the like. Figure 1 In another example not shown in the figure, the output device can be a presentation device such as a display, light source, speaker, vibration motor, etc., which can be used to generate a sensory-perceptible output signal, which can present an inferred state or a result derived from the inferred state (such as a fault diagnosis or any other type of derived result), for example for use in guidance, navigation, or other types of control of a physical system.

[0056] The system 100 is further shown to include a processor subsystem 160 configured to iteratively infer a state of the physical system based on sensor data 122 accessed via the input interface 120, and to provide output data 142 to the output device 40 to enable control of the physical system 60 based on the inferred state. For this purpose, the processor subsystem 160 can be configured to: obtain sensor measurements; obtain an initial inference of the state using a mathematical model, the mathematical model representing a prior knowledge-based modeling of the state as a function of the sensor measurements and previously inferred states; apply a learned model to the initial inference of the state and the sensor measurements, wherein the learned model has been learned to minimize the error between the initial inference provided by the mathematical model and the ground truth, and provide a correction value as an output for correcting the initial inference of the state of the mathematical model; and obtain a current inferred state by combining the initial inference of the state with the correction value.

[0057] The system 100 is further shown as including a data storage interface 180 for accessing a data storage device 190, which can be a volatile or non-volatile type of data storage device and can be used to temporarily or permanently store data used by the processor subsystem 160, including but not limited to a data representation 192 of a mathematical model and a data representation 194 of a learned model.

[0058] refer to Figure 2 and Figure 3 , various details and aspects of the operation of system 100, including optional aspects, will be further clarified.

[0059] In general, the system may be embodied as a single device or apparatus (such as a workstation or server) or in the single device or apparatus. The server may be an embedded server. The device or apparatus may include one or more microprocessors that execute appropriate software. For example, the processor subsystem may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such CPUs and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory (e.g., a volatile memory such as RAM or a non-volatile memory such as flash memory). Alternatively, the processor subsystem of the system may be implemented in a device or apparatus in the form of programmable logic, for example as a field programmable gate array (FPGA). In general, each functional unit of the system may be implemented in the form of a circuit. The system may also be implemented in a distributed manner, for example, involving different devices or apparatuses (such as distributed local or cloud-based servers). In some embodiments, the system may be part of the physical system itself, and / or may represent a control system configured to control the physical system.

[0060] Various embodiments are conceivable for iteratively inferring the state of a physical system based on sensor data while using a "hybrid" approach in the iterative inference, the "hybrid" approach involving a learned model and a prior knowledge-based ("non-learned") mathematical model. The following example describes iterative inference by modeling iterative inference as a directed graphical model that uses an iterative message passing scheme over the edges of the directed graphical model. However, it will be appreciated that such a message passing scheme can be used to explain iterative inference, but the actual implementation of iterative inference can be implemented in various other ways, such as on the basis of similar mathematical concepts. In particular, as also discussed elsewhere, what is referred to as a "prior knowledge message" hereinafter can represent a message passing-based representation of a mathematical model, while a "learned message" can represent a message passing-based representation of a learned model.

[0061] Furthermore, in the following examples, a recursive neural network is used as the learned model, which in this example is a graph neural network (GNN) that includes a gated recursive unit (GRU) to establish recursion in the graph neural network. However, it will be appreciated that other types of recursive neural networks may also be used, or in general any other type of learned model that provides functionality as described. The inference itself is based on a mathematical model that includes a transition model part and a measurement model part, the transition model part models the conditional probability of the state to be inferred given a previous inferred state, the measurement model part models the conditional probability of sensor measurements given the state to be inferred, and the probability distribution of the mathematical model is assumed to be linear and Gaussian. As such, the following can be considered to be a "hybrid" approach for Kalman filtering by incorporating a learned model as described.

[0062] As indicated above, iterative inference can be modeled as a directed probabilistic graphical model (also referred to simply as a “generative model” hereinafter) using a message passing scheme in which nodes of the graphical model can send and receive messages to infer states An estimate of . A hybrid approach is described below, where the message derived from the generative graphical model is combined with the learned message, in short:

[0063] 1. Prior knowledge messages: These messages can be derived from the generative graphical model (e.g., equations of motion from a physics model).

[0064] 2. Learned messages: These messages can be learned using graph neural networks, which can be trained to reduce inference errors on annotated data combined with prior knowledge messages.

[0065] Hidden Markov Model

[0066] Figure 2 As background, a Hidden Markov Model is shown, in which a set of unobservable variables At each time step The set of observable variables from which one might want to infer the state of a process is given by People can is expressed as a probability distribution over the hidden states given an observation. It might be desirable to find which states Maximize this probability distribution. More formally:

[0067]

[0068] Under the Markov assumption, these variables can follow a graphical model structure, where i) the transition model can be obtained by the transition probability and ii) the measurement model can be described by The two distributions can be described for all is fixed. The resulting graphical model can be expressed using the following equation:

[0069]

[0070] Well-known methods for inference problems in this graphical model are Kalman filters and smoothers. In the Kalman filter, both the transfer and measurement distributions are assumed to be linear and Gaussian: the prior knowledge one may have about the process can be encoded in the linear transfer and measurement processes, and the uncertainty in the prediction relative to the real system can be modeled by Gaussian noise:

[0071]

[0072]

[0073] Here , Can come from a Gaussian distribution , ,in , are the linear transfer and measurement functions respectively. If one can infer from them The process is actually Gaussian and linear, then with the correct parameters ( ) will be able to infer the best state estimate. However, the real world is often nonlinear and complex, so assuming that the process is linear can be a strong limitation.

[0074] To model the complexities of the real world, these complexities can be learned from data (also called "ground truth" or "training data") by a learnable model such as a neural network. The following hybrid approach, also called Graph Recursive Inference Network (GRIN), combines knowledge from a generative model (e.g., physics equations) with corrections learned from the training data using a neural network. Experiments have shown that the hybrid approach outperforms prior knowledge-based approaches and also neural network approaches for low and high data regimes, respectively. In other words, the hybrid approach benefits from the inductive bias in the small data limit and also the high capacity of neural networks in the large data limit. The hybrid approach can interpolate well between these different regimes.

[0075] Prior knowledge message

[0076] In order to limit the prior knowledge information, Methods Equation (2) can be extended to a probabilistic graphical modeling framework. This can be interpreted as estimating the state The iterative optimization process of the maximum likelihood value of . The recursive update for each successive estimate of can be given by:

[0077]

[0078] Simplifying Eq. (5) from Eq. (2) for a hidden Markov process allows for each inferred node to be Outputs three input messages:

[0079]

[0080]

[0081] The three messages can be obtained by calculating the three derivatives according to equations (7), (8), and (9). It is usually assumed that the transfer and measurement distributions , is linear and Gaussian, resulting in a Kalman filter model. Next, when these linear and Gaussian functions are assumed as in (3) and (4), an expression for the prior knowledge message can be provided:

[0082] .

[0083] Add learned message

[0084] This section describes how to add a learned signal to the recursive operation. Instead, the model is transformed into a hybrid model or algorithm, where Figure 4 The inference iterations of the hybrid model are shown. The resulting equation is:

[0085]

[0086] Signal A graph neural network (GNN) can be constructed to aggregate information between nodes. For example, a gated recurrent unit (GRU) can be added to the message passing operation to make the message passing operation recursive:

[0087]

[0088] For each time step t , one can represent the hidden states of the nodes of the directed chain graph Keep track. If two nodes In the time dimension t If they are continuous, they can be connected. For example, it can be a 2-layer graph neural network, which aggregates neighborhood information into messages for each node. In. GRU can receive the message at the input gate: ; people can mark The concatenation of the prior knowledge messages (7), (8), and (9); and the observed values Based on this input data, Can update hidden state Finally, MLP The hidden state Mapping to the correction signal .

[0089] Equation (13) can thus be expressed in a simple recursive form as a hybrid model or algorithm (also known as a GRIN (Graph Recursive Inference Network) model), where the update can be done by two contributions: : One that depends on a previously known equation - , and another—— , which has been learned.

[0090] During training of the learned model, the loss function can be the inferred state With the basic facts To provide early feedback, the loss function can be calculated at each iteration using a weighted sum that emphasizes later iterations, , more formally:

[0091] .

[0092] In some examples, the ground truth may consist only of the inferred state For example, in some localization tasks, the state may describe the position and speed of the process, while only the ground truth may be available for the position. In such a case, the loss can be calculated using the portion of the state included in the ground truth. The resulting loss can be the following:

[0093]

[0094] in is a matrix whose rows are one-hot encoded vectors from the vectors contained in the ground truth Select the quantity.

[0095] Training the learned model can include three main steps. First, each Can be initialized to the initial value. To speed up convergence, you can choose to maximize For example, in the context of trajectory estimation, The position value can be set to the observed position Secondly, the hyperparameters of the prior knowledge model can be tuned as they would be done with a Kalman filter, which are typically the variances of the measurement and transfer Gaussian distributions. Finally, the learned model can be trained using the loss functions (15), (16) mentioned above.

[0096] In three different datasets for trajectory estimation, namely, a linear synthetic dataset, a nonlinear chaotic system (Lorenz attractor), and a real-world positioning system (Michigan NCLT dataset), it has been found that the GRIN model efficiently combines prior knowledge messages with learned messages, thereby outperforming either learned inference or graphical inference alone for different data systems.

[0097] Figure 4 A computer-implemented method 200 is shown for enabling control of a physical system based on a state of the physical system inferred from sensor data. The method 200 may correspond to the method described in reference Figure 1 And the operation of the system described elsewhere. However, this is not a limitation, as the method can also be implemented by another system, device or apparatus.

[0098] The method 200 may include accessing 210 sensor data representing sensor measurements associated with a state of the physical system using an input interface in an operation entitled “Accessing Sensor Data”. The method 200 may further include iteratively inferring the state of the physical system based on the sensor data in an iteration 220 of iterative inference by: obtaining 230 sensor measurements from the sensor data in an operation entitled “Obtaining Sensor Measurements”; obtaining 240 an initial inference of the state using a mathematical model in an operation entitled “Obtaining Initial Inferences from Mathematical Model”, the mathematical model representing prior knowledge-based modeling of the state as a function of the sensor measurements and previously inferred states; applying 250 a learned model to the initial inference of the state and the sensor measurements in an operation entitled “Obtaining Correction Values ​​from Learned Model”, wherein the learned model has been learned to minimize an error between the initial inference provided by the mathematical model and a ground truth, and providing a correction value as an output for correcting the initial inference of the state of the mathematical model; and obtaining 260 a current inferred state by combining the initial inference of the state with the correction value in an operation entitled “Combining Initial Inferences with Correction Values”. The method 200 may further include, in an operation entitled "Providing output data to an output device", providing 270 output data to an output device used in controlling the physical system using the output interface to enable control of the physical system based on the current inferred state. It will be appreciated that, in general, Figure 4 The operations of method 200 may be performed in any suitable order (eg, serially, concurrently, or a combination thereof), subject to, where applicable, a particular order being necessitated, for example, by input / output relationships between the respective operations.

[0099] The method can be implemented on a computer as a computer-implemented method, dedicated hardware, or a combination of both. Figure 5 As illustrated in , instructions for a computer (e.g., executable code) can be stored on a computer readable medium 300, for example, in the form of a machine-readable physical mark sequence 310 and / or as a sequence of elements having different, for example, electrical, magnetic, or optical properties or values. The executable code can be stored in a temporary or non-temporary manner. Examples of computer readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 5 An optical disk 300 is shown. Alternatively, the computer readable medium 300 may include transient or non-transitory data 310 representing a learned model as described elsewhere in this specification.

[0100] Examples, embodiments or optional features, whether or not indicated as non-limiting, should not be construed as limiting the invention as claimed.

[0101] It should be noted that the above-mentioned embodiments illustrate but do not limit the present invention, and those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be interpreted as limiting the claims. The use of the verb "comprise" and its morphological changes does not exclude the presence of elements or stages other than those stated in the claims. The article "one" or "an" in front of an element does not exclude the presence of multiple such elements. Expressions such as "at least one of ..." when in front of an element list or element group indicate that all elements or any subset of elements are selected from the list or group. For example, the expression "at least one of A, B, and C" should be understood to include: only A; only B; only C; both A and B; both A and C; both B and C; or all of A, B, and C. The present invention can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a device claim that lists several components, several of these components can be embodied by the same hardware item. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A system (100) for enabling control of a physical system based on a state of the physical system inferred from sensor data, comprising: - an input interface (120) for accessing sensor measurements (y1:y2) associated with said state of the physical system (60) T )’s sensor data (122); - an output interface (140) to an output device (40) used in the control of the physical system; - a processor subsystem (160) configured to iteratively infer the state of the physical system based on the sensor data by iteratively: - Use the input interface to obtain sensor measurements (y T ); - obtaining an initial inference of the state using a mathematical model (192) that represents the sensor measurements (y T ) and the previous inferred state (x T )'s function to model the state based on prior knowledge; - Applying the learned model (194) to the initial inference of the state and the sensor measurements (y T ), where the learned model has been learned to minimize the error between the initial inference provided by the mathematical model and the ground truth, and provides a correction value (e T+1 ) as an output for correcting an initial inference of said state of said mathematical model; as well as - by comparing the initial inference of the state with the correction value (e T+1 ) to obtain the current inference state (x T+1 ); in: The learned model is a recurrent neural network (RNN); and The processor subsystem is further configured to: maintaining and communicating a hidden state of the recurrent neural network between iterations of iterative inference of the state of the physical system; and The output interface is used to provide output data (142) to the output device to enable the implementation of the current inference state (x T+1 )Control of physical systems.

2. The system (100) of claim 1, wherein the processor subsystem (160) is configured to: - Using the input interface (120) to obtain the time series (y1:y T ); - by using the time series (y1:y T ) and the time series of the previous inferred state (x1:x T ) as input to the mathematical model (192) to obtain an initial inference of the state; - Applying the learned model (194) to the initial inference of the state and the time series of sensor measurements (y1:y T ) to obtain the current correction value (e T+1 ), the current correction value (e T+1 ) and the previous calibration value (e1:e T ) together form the time series of correction values ​​(e1:e T+1 );and - by comparing the initial inference of the state with the time series of correction values ​​(e1:e T+1 ) to obtain the current inference state (x T+1 ).

3. The system (100) according to any one of claims 1 to 2, wherein the learned model is a graph neural network (GNN), which includes a gated recurrent unit (GRU) to establish recurrence in the graph neural network.

4. The system (100) according to any one of claims 1 to 3, wherein the mathematical model provides as sensor measurements (y T ) and the previous inferred state (x T )'s function to provide a physics-based modeling of the state.

5. The system (100) according to any one of claims 1 to 4, wherein the mathematical model comprises a transition model part and a measurement model part, wherein the transition model part is a conditional probability (p(x)) of a state to be inferred given a previous inferred state. T :x T-1 )) is used to model the conditional probability (p(y t :x T )) for modeling.

6. The system (100) of claim 5, wherein the processor subsystem (160) is configured to iteratively infer the state by Kalman filtering by assuming that the probability distribution of the mathematical model is linear and Gaussian.

7. The system (100) according to any one of claims 1 to 6, wherein: The processor subsystem (160) is configured to implement iterative inference of the state of the physical system as a directed graphical model and by using an iterative message passing scheme over the edges of the directed graphical model.

8. The system (100) according to any one of claims 1 to 7, wherein the output device (40) is an actuator associated with the physical system (60), and wherein the system is configured to output the output device (40) to the actuator by providing the actuator with a current inferred state (x) of the physical system. T+1 )'s control data (142) to control the physical system.

9. The system (100) according to claim 8, wherein: The system is one of: -Vehicle control systems; -Robot control system; - Manufacturing control systems; and -Building control systems.

10. The system (100) according to any one of claims 1 to 7, wherein: The output device (40) is a presentation device, and wherein the system is configured to enable an operator to display a current inferred state (x) of the physical system by causing the presentation device to display a current inferred state (x) of the physical system. T+1 ) generates sensory output signals to control physical systems.

11. The system (100) according to claim 10, wherein: The system is configured to take into account the current inferred state of the physical system (x T+1 ) indicates a fault in a physical system, causing the generation of a sensory perceptible warning signal.

12. A computer-implemented method (200) for enabling control of a physical system based on a state of the physical system inferred from sensor data, comprising: - using an input interface to access (210) sensor measurements (y1; y2) associated with said state of the physical system T )’s sensor data; - iteratively inferring (220) the state of the physical system based on the sensor data by iteratively: - obtain (230) sensor measurements (y) from sensor data T ); - obtaining (240) an initial inference of the state using a mathematical model representing the sensor measurements (y T ) and the previous inferred state (x T )'s function to model the state based on prior knowledge; - Applying (250) the learned model to the initial inference of the state and the sensor measurements (y T ), where the learned model is a recurrent neural network (RNN) that has been learned to minimize the error between the initial inference provided by the mathematical model and the ground truth, and to provide a correction value (e T+1 ) as an output for correcting an initial inference of said state of said mathematical model; and - by comparing the initial inference of the state with the correction value (e T+1 ) are combined to obtain (260) the current inference state (x T+1 ); - maintaining and transferring a hidden state of the recurrent neural network between iterations of iterative inference of said state of the physical system; as well as - using the output interface to provide (270) output data to an output device used in the control of the physical system to enable implementation based on the current inferred state (x T+1 )Control of physical systems.

13. A computer readable medium (300) comprising transitory or non-transitory data (310) representing instructions arranged to cause a processor system to perform a computer implemented method according to claim 12.

14. A computer readable medium (300) comprising transient or non-transient data (310) representing a learned model, the learned model being learned to minimize the error between i) an initial inference of the state of a physical system using a mathematical model and ii) a ground truth, and providing a correction value (e T+1 ) as an output for correcting the initial inference of the state of the mathematical model, the mathematical model representing the previously inferred state (x T ) and sensor measurements associated with the state of the physical system (y T ) based on prior knowledge of the state, wherein the learned model is a recurrent neural network (RNN); and Instructions that are executable by a processor and that, when executed by the processor, cause the processor to perform the following operations: iteratively infer a state of a physical system based on sensor data, and maintain and transfer a hidden state of a recurrent neural network between iterations of iterative inference of the state of the physical system.

Images