Multimodal state estimation with masked sensor measurements

Abstract

Computing unit (1) for estimating at least one component of a system state of an observed system (2), wherein the computing unit (1) is designed to receive data streams from measurements of at least two different sensors (3, 4, 5) and to use the data streams of the sensors (3, 4, 5) or data products derived from sensor measurements as input variables of a state estimator implemented in the computing unit (1) to determine a current estimate of the component of the system state, wherein a mathematical model of the observed system (2) is implemented in the computing unit (1) as a system model and a respective further mathematical model of the respective sensor (3, 4, 5) is implemented as a respective measurement model, characterized in that a predefined hierarchy of the sensors (3, 4, 5) is stored in the computing unit (1) according to a predefined priority of their measurements, and wherein the computing unit (1) is designed toto mask the measurement of at least one of the sensors (3, 4, 5) other than the one at the top in the hierarchy with a mask determined by the null space of the measurement model of at least the sensor (3, 4, 5) immediately above it in the hierarchy, so that measurements of at least two sensors (3, 4, 5) are complementarily included in the state estimator, the mask being a projection into the null space of the measurement model of at least the sensor (3, 4, 5) immediately above it in the hierarchy.

Description

[0001] The invention relates to a computing unit for estimating at least one component of a system state of an observed system, as well as a method for estimating at least one component of a system state of an observed system.

[0002] In technical systems, quantities of interest are often measured directly by a sensor, and occasionally the measured signal is further processed, for example, by frequency filtering. If quantities of interest cannot be measured directly, or if an alternative method of recording the respective quantity is required, it can sometimes be derived from other measured quantities. In so-called sensor fusion, the results of sensors measuring the same quantity, which may be based on different principles, can be fused into a common value, or a measurement and an estimate of the same quantity can be fused into a common value. The common value is therefore always itself an estimate, not a direct measurement of the quantity.

[0003] Often, a quantity defined as a system state, such as a vehicle's speed, needs to be quantified as realistically as possible for further technical use. For example, if i) a vehicle speed derived from a measurement of the angular velocity of a vehicle's wheel, converted using a wheel radius, and ii) a vehicle speed determined by a satellite-based positioning system are fused together, an estimate of the vehicle's speed is obtained as a result of the data fusion. This estimate generally indicates the actual vehicle speed more reliably and accurately than one of the (converted) measured values ​​alone. The use of different measured quantities in sensor fusion (as in the above example) is referred to as multimodal sensor fusion.

[0004] In the technical application of sensor fusion systems, attempts are often made to estimate the state values ​​of a large number of state variables of the physical system simultaneously, rather than a single state variable of an observed physical system. This is called multivariate state estimation. If a system state comprises multiple variables, each component of the system state can be directly measured by a sensor—if technically possible. However, only a subset of the variables can be directly measured, and the remaining components of the system state can be estimated based on the measurements. If the combination of all measurements from different sensors allows conclusions to be drawn about the complete system state, the system is said to be fully observable.

[0005] Known sensor fusion methods use probabilistic approaches that weight the various measured variables based on their uncertainties. This typically results in a trade-off between the desired computational efficiency and the most realistic representation of the underlying probabilities. State-of-the-art methods, such as the Kalman filter and its derivatives, are fundamentally based on the assumption that all incoming sensor data is normally distributed; they therefore consider the sensor data equally, simply weighted according to their uncertainties. Furthermore, it is a prerequisite that the inherent system dynamics (as a system model) and the sensor measurements (as a measurement model) can be described mathematically precisely. With each violation of one of these conditions, further deviations from the ideal estimate of the variable of interest must be accepted.Other methods well known in the state of the art, such as the particle filter, are able to take non-normally distributed information into account, but require significantly higher computational resources.

[0006] A Kalman filter is typically model-based, using a system model and a measurement model. These models are inherently error-prone.

[0007] The state-of-the-art method "Linearly Constrained Kalman Filtering under Model Mismatch" allows the influence of (explicitly specified) parameters on the state estimation in the system model and the measurement model to be minimized by "hiding" their influence. An example of "Linearly Constrained Kalman Filtering under Model Mismatch" can be found in: "Ortega Espluga, Lorenzo & Vilà-Valls, Jordi & Chaumette, E. & Pages, Gael & Vincent, Francois. (2020). Robust Tracking under Measurement Model Mismatch via Linearly Constrained Extended Kalman Filtering. 10.1109 / CDC42340.2020.9303842."

[0008] Other approaches, such as robust filtering, are methods that make state estimators more robust against measurement outliers that lie outside the expected probability distribution. Robust filtering is discussed, among others, in this publication: "Daniel Medina, Robust GNSS Carrier Phase-based Position and Attitude Estimation, Ph.D. thesis, Universidad Carlos III de Madrid, 2022."

[0009] The state of the art primarily aims to fuse all sensor data together, weighted according to the modeled measurement uncertainty. If sensor data occurs that qualitatively correspond significantly less to the assumptions made (e.g., because they are not normally distributed and exhibit a bias) than that of another sensor, this can negatively impact the entire state estimation and potentially lead to a complete failure of the state estimation. At least the current state of the art allows the influence of individual faulty model parameters, whether in the sensor models or in the process models, on the sensor fusion to be estimated or excluded. However, specific parameters must be explicitly defined for this, and the number of possible parameters is limited to enable successful state estimation.

[0010] In the publication “Optimal sequential Kalman filtering with cross-correlated measurement noises” by the authors Chuanbo Wen, Yunze Cai, Chenglin Wen, Xiaoming Xu, from Aerospace Science and Technology, Volume 26, Issue 1, 2013, Pages 153-159, ISSN 1270-9638, https: / / doi.org / 10.1016 / j.ast.2012.02.023, an optimal sequential decentralized filtering algorithm for discrete time-varying linear control systems with cross-correlated noise is shown.

[0011] The object of the invention is therefore to improve the estimation of at least one component of a system state of a system under consideration on the basis of several input data based on measurements.

[0012] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0013] A first aspect of the invention relates to a computing unit for estimating at least one component of a system state of an observed system, wherein the computing unit is designed to receive data streams with measurements from at least two different sensors and to use the data streams of the sensors as input variables of a state estimator implemented in the computing unit to determine a current estimate of the component of the system state, wherein a mathematical model of the observed system is implemented in the computing unit as a system model and a respective further mathematical model of the respective sensor is implemented as a respective measurement model, characterized in that a predefined hierarchy of the sensors is stored in the computing unit according to a predefined priority of their measurements, and wherein the computing unit is designed toto mask the measurement of at least one of the sensors other than the one at the top in the hierarchy with a mask determined by the null space of the measurement model of at least the sensor immediately above it in the hierarchy, so that measurements of at least two sensors are complementarily included in the state estimator, the mask being a projection into the null space of the measurement model of at least the sensor immediately above it in the hierarchy.

[0014] The null space refers to the measurement model of the higher-level sensor that actually flows into the system state, which can, for example, contain its own pre-calculation steps or masking.

[0015] The system state describes a state, defined as such, of an observed system. The observed system is preferably a physical system, but can also be a hybrid system with physical components of interest and abstract or virtual components, such as a constructed energetic state component with physical and artificial parameters, or similar.

[0016] An example of a physical system under consideration is an aircraft whose states are of interest: speed, position, altitude, and orientation in terms of attitude angles relative to the Earth. In this example, the system state is vectorial, meaning the entire system state is composed of a multitude of components. However, it is also conceivable that only a single scalar component of a system under consideration is defined as the system state.

[0017] The task of the computing unit is to estimate at least one component by deriving it from at least two measurements from independent and different sensors, because it is either not measurable at all, or if it is directly measurable, it is nevertheless advisable to carry out a sensor fusion to estimate the component, since the result of the sensor fusion is more reliable and / or of higher quality with regard to the representation of reality than a single measurement from a sensor.

[0018] For both use cases, the state estimator is implemented in the computing unit, for example, a Kalman filter or its derivative. The input variables of the state estimator implemented for sensor fusion are always the data streams from at least two different sensors. In principle, any number of measurements from different sensors can be used as input variables for the state estimator, including previously processed measurements, such as (frequency) filtered, reconstructed measurements, or other observations such as the results of previously executed filters and preprocessing. The data streams from the sensor measurements should therefore be understood as broader terms than just original measurements; rather, they are sensor results that may already have been processed.The data streams result from the fact that measured values ​​are transmitted to the computing unit, preferably via digital signals with a certain repetition frequency, to be used as input variables in the state estimator. The computing unit is therefore also designed to execute the state estimator in a loop, i.e., repeatedly, for example, at a frequency of 200 Hz.

[0019] The system to be measured is preferably described mathematically in the state estimator using a state representation, as is generally state-of-the-art. The characteristics of the sensors themselves are also described mathematically for each type of sensor, thereby obtaining the respective measurement model. If the respective mathematical relationship of the measurement model is linear, the description can be achieved using the measurement model as a matrix-vector representation with a measurement matrix; if non-linear relationships exist, the dynamics of the sensor in question are preferably linearized around a so-called operating point in order to achieve a matrix-vector representation that includes a measurement matrix. This procedure corresponds to the state of the art, for example for Kalman filtering or extended Kalman filtering, and all other attributable derivatives. The state estimator therefore generally has a large number of measurement matrices.

[0020] The corresponding null space can be derived from the respective (possibly modified) measurement model. This null space is used in the form of a mask for the next worse sensor in the hierarchy. For this next worse sensor, only the measurement in the state estimator that lies within this null space is used. This null space is generally generated based on the respective measurement model of one or more of the higher-quality sensor measurements in order to create a mask for the lower-quality sensor measurements. After masking, the different measurements become complementary to one another, with the measurement of the lower quality merely supplementing the measurement of the higher quality. If the masking and combination of all obtained values ​​is combined with known filter approaches, a supplementary filter is obtained.The masking is therefore carried out in particular by a projection of measured values ​​into a subspace of linear algebra, which is defined by the characteristic of the higher measurement.

[0021] The hierarchy of sensors is preferably defined manually, whereby it is the user's responsibility to define the measurement priorities in such a way that measurements of higher quality are reflected in a correspondingly high position of a sensor in the hierarchy. However, the quality of the measurements is not necessarily the sole criterion for determining priority; other criteria can also be used.

[0022] An advantageous effect of the invention is that it is possible to combine measured values ​​from different sensors in a state estimator, which can be a multivariate state estimator, through masking, thus obtaining an improved state estimate of all components of interest. If a multimodal measurement system has sensors of different quality, it is therefore possible to mask sensor measurements of lower quality in such a way that higher-quality sensor measurements are not degraded, while ensuring the greatest possible observability of the system.

[0023] This approach therefore allows for the masked combination of various measurements of different origins and of different quality for a state estimation, resulting in an improved state estimate than would be possible with conventional methods. Low quality here specifically means that measurements exhibit significant measurement noise and / or are corrupted by systematic errors. Improved in this case means that all information from a high-quality sensor is used, but all low-quality sensor measurements are masked so that they only cover those dimensions that are instantly unobservable with the high-quality sensor – thus supplementing the instantaneous observability of the system state. This creates a supplemented measurement of the system state, which limits the influence of low-quality sensors to what is necessary while still obtaining a complete measurement of the system state.

[0024] To determine the respective null space, the kernel of a measurement matrix from the measurement model of a higher-order sensor measurement is preferably used to generate a mask for the lower-order sensor measurements. After masking, the different measurements become complementary to each other, with the lower-order measurement merely supplementing the other measurement. This can thus be referred to as a "supplementary / complementary state estimation."

[0025] For example, a system state with six components needs to be estimated. If a first sensor measures the first three of these components, and a second sensor measures five of these components, of which the first three are identical to those of the first sensor, a third sensor can measure the final component. However, due to the overlap of the measured components, lower-quality variables can be masked from the input variables for the state estimator. Only the two components not detected by the second sensor, as well as the measured component of the third sensor, cannot be omitted.

[0026] The procedure for deriving the state space model and determining the masks from the corresponding null spaces of the measurement matrices can be done as follows:

[0027] Consider a discrete state-space model (SSM). This model describes the evolution of the (vector) system state x over a certain time period using a process model f(·) and its relationship to the measurements, generally referred to as observations, y using an observation model h(·): xk=f(xk−1, uk−1, wk−1) yk=h(xk)+ηk Here, u describes the system input, and w k-1 ≈ N(0, Q k-1 ) with N(iHk) the respective corresponding null space, and for the (nominal) measurement noise η k ≈ N(0, R k ).

[0028] Using the "extended Kalman filter (EKF) framework" as a state estimator, the above system is linearized around the current estimate of x. For the linear observation, this results in: yk=Hkxk+ηk

[0029] Here, Hk=∂h∂x|xk* the measurement Jacobian matrix, linearized around the predicted a-priori state xk*.

[0030] Furthermore, the conditions x∈V from the state definition domain and y∈U from the measurement domain. Here, N denotes the dimension of the state and M the dimension of its observation. Typically, the following is assumed: x ∈ R N and y ∈ R M .

[0031] The following shows how the primary measurement is supplemented: It is assumed that the set of simultaneous state observations y k considered measurements various individual observations i y k which are determined by I different sensors, where i = 0, ..., I-1. The corresponding linearized measurement Jacobian matrices i H k are linear mappings V→u(Hik) the same state space into several different measurement spaces ui with ui⊆Ui. In the following example approach of supplementary filtering, all sensors for measurements were manually sorted hierarchically according to their expected measurement priority, in this case measurement quality. A primary measurement 0 y k should be chosen for which the highest measurement quality is assumed, but which can only observe a part of the entire state vector x of interest. The corresponding linearized measurement Jacobian matrix 0 H k therefore has insufficient column rank with rank( 0 H k ) < N. Due to the insufficient column rank of 0 H k there exists a non-trivial null space of the kernel N(H0k) to measurable space N⊥(H0k). Therefore, similar to the publication: "Kuo, Joseph et al. (Mar. 2022). "Computing a projection operator onto the null space of a linear imaging operator: tutorial". In: J. Opt. Soc. Am. A 39.3, pp. 470-481. DOI:10.1364 / JOSAA.443443. URL: https: / / opg.optica.org / josaalabstract.cfm?URI =josaa-39-3-470" projection operators can be introduced for masking: Pinull:V→N(Hik), Pimeas:V→N⊥(Hik)

[0032] It should be noted that P meas reduces the definition space of system states to the measurable space, but H reduces the definition space to the measurement space. Therefore, ui⊆Ui and N⊥(H0k)⊆Vi. Second measurements 1 y k are used in addition to the primary measurements, and all state dimensions that are already observed are masked by 0 y k .

[0033] The null space projectors of the measurement matrices i H kare used to derive the projection operators for masking. However, this is not the only way to generate a supplementary filter. Any suitable projection that generates complementary subspaces can be used with the present approach; their definition depends on relevant design criteria.

[0034] There are several ways to bring the projectors into the null space N / A) and the corresponding measurable space N⊥(A) a rectangular matrix A. The projection operators can be calculated directly, both by means of singular value decomposition and via the pseudoinverse, which, according to the publication “Klema, V. and A. Laub (1980). The singular value decomposition: Its computation and some applications. In: IEEE Transactions on Automatic Control 25.2, pp. 164-176. DOI : 10.1109 / TAC.1980.1102314”, are related as follows: Pnull(A)=I−A+A=V2V2T[n×n] Pmeas(A)=A+A=V1V1T[n×n]

[0035] This is followed by the singular value decomposition of the real matrix A with A=U∑VT=[U1U2][S000][V1TV2T] where U1, U2, V1, V2 are orthogonal bases of the projective subspaces of A (cf. the above publication by Klema et al.). According to Klema et al., the image of U1 defines the measurement space U(A) and the image of V2 the null space N / A) If A is used in a linear map y = Ax, the results can be represented with respect to the basis of the projective subspace U. This is done by taking advantage of the fact that, due to orthogonality, U T U = UU T = I and V T V = VV T I, from which we obtain: (UUT)y=A(VVT)x, and UTy=UTAV(VTx)=∑(VTx).

[0036] If the singular value decomposition is interpreted as a composition of a rotation V T, dilation Σ and a second rotation U, the final rotation suppresses the linear transformation of the above expression and instead returns the result with respect to the basis U. Based on the notation of singular value decomposition introduced above, it can be written: [U1TU2T]y=[S000][V1TV2T]x,⇔y˜:=U1Ty=SV1Tx, where all rows referring to the zero-valued singular values ​​are removed.

[0037] The method of weighted generalized inverses is an alternative approach and can be found in the publication: “Doty, Keith L., Claudio Melchiorri, and Claudio Bonivento (1993). A Theory of Generalized Inverses Applied to Robotics. In: The International Journal of Robotics Research 12.1, pp. 1-19. DOI: 10.1177 / 027836499301200101.”, whereby the generalized inverse A # In the case of unit weights, the Moore-Penrose pseudo-inverse, or A +, can be obtained. Under certain circumstances, the well-known QR decomposition can also be used.

[0038] The masking is achieved by projecting the state space V into the null space of the measurement 0 by 0 P null This creates 0 P null in the state space V; the aim, however, is to carry out the measurements directly in their measurement spaces 1U The masking for the first supplementary measurement in its measurement definition range iU is therefore: 1Φk=1HkP0null1Hk#.

[0039] It will now be y k = h(x k ) + η k multiplied by the masking and thus obtained: 1yk*=1HkP0null1Hk#1Hkxk+1HkP0null1Hk#1ηk,=1HkP0nullP1measxk+1HkP0null1Hk#1ηk, where 1yk*=Φ1ky1k is the first masked and thus supplementary measurement. For further measurements, the respective masking contains all previous null space projections by Φik=iHkP0nullP1null…Pi−1null1Hk#; This results in: iyk*:=Φikyik, and: H*ik:=ΦikiHk until i Φ k reaches a zero rank. Finally, the measurement noise model is masked. As mentioned at the beginning, it is simplified to assume that all i η are normally distributed with a mean of zero. The masked covariance is therefore: R*ik=E(η*ikη*ikT)=ΦikηikηikTΦikT=ΦikiRΦikT

[0040] In the following, the minimal representation is considered, and for this purpose, first the measurement matrix with full rank: While previously the masked supplementary observation model was obtained, the matrices H*ik:=ΦikiHk potentially have too low a row rank due to the subspace projections of the introduced masks. However, measurement matrices with full row ranks can be obtained by changing the basis for all masked measurements. This is achieved by iyk*:=Φikyik using (UU T )y = A(VV T )x and [U1TU2T]y=[S000][V1TV2T]x,⇔y˜:=U1Ty=SV1Tx together with the singular value decomposition of i Φ k is rewritten to obtain: 1y˜k*=U1,ΦikT1yk*=SΦikV1T,Φiky

[0041] To obtain the minimum measurement matrices and the minimum measurement noise covariance matrices, i Φ k through SΦikV1T,Φik Now all the individual elements can be combined to obtain a new observation model, where y k = H k x k + n k by ỹk = H k x k + η̃ k is replaced, where the vectors and matrices contain the supplementary information as follows: y˜k=[y0kSΦikV1T,Φikyk⋯SΦikV1T,Φikyk], and: H˜k=[H0kSΦ1kV1T,Φ1kHk⋯SΦikV1T,ΦikHk]

[0042] The supplementary measurement noise of a sensor is then: R˜*ik=SΦikV1T,ΦikRikV1,ΦikSΦik, and thus: R˜*k=[R˜*1kR˜*2k⋯R˜*ik] for the complete measurement vector.

[0043] The supplementary Kalman filter is shown below: A standard Kalman filter calculates the estimates of the vector state and its corresponding covariance matrix P as follows (see the publication: Simon, Dan (2006). Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches. Wiley-Interscience. ISBN: 9780471708582), where the predictive step is: x^k−=Fk−1x^k−1++Gk−1uk−1, Pk−=Fk−1Pk−1+Fk−1T+Qk−1.

[0044] The correction update step is: Kk=Pk−HkT(HkPk−HkT+Rk)−1 x^k−1+=x^k−1−+Kk(yk−Hkx^k−1−), Pk+=(I−KkHk)Pk−(I−KkHk)T+KkRkKkT

[0045] In the case of the complementary filter, the modified ỹ k , H k , R k out of... y˜k=[ 0ykS1ΦkV1T,1Φkyk ⋯SiΦkV1T,iΦkyk],H˜k=[ 0HkS1ΦkV1T,1ΦkHk ⋯SiΦkV1T,iΦkHk]and R˜k=[1R˜k*2R˜k*⋯iR˜k*] used.

[0046] According to the invention, the mask is a projection into the null space of the measurement model of at least the sensor immediately above it in the hierarchy.

[0047] According to a further advantageous embodiment, the mask is a projection onto the core of the measurement matrix of the measurement model of at least the sensor immediately above it in the hierarchy.

[0048] According to a further advantageous embodiment, the state estimator is a Kalman filter or a derivative of a Kalman filter. A derivative of a Kalman filter is, for example, an extended Kalman filter or an information filter.

[0049] According to a further advantageous embodiment, the computing unit is configured to apply the respective mask used for the measurement to a covariance matrix of the Kalman filter or the derivative. The covariance matrix is ​​part of an estimation algorithm, such as a Kalman filter, which is based on the assumption that a respective measured value is an average of the measurement combined with an uncertainty value. This statistical perspective requires the implementation of a covariance matrix, which is also tailored to the measured values ​​using the corresponding mask.

[0050] According to a further advantageous embodiment, the computing unit is designed to mask the measurement of at least one sensor other than the one at the top in the hierarchy with a cumulative mask by determining the cumulative mask from each null space of the respective measurement model (in particular from each kernel of a measurement matrix) of all sensors located above it in the hierarchy, so that only those measurements of a sensor are included in the state estimator which are not made by any sensor higher in the hierarchy.

[0051] In this case, the masks accumulate starting from the topmost sensor in the hierarchy and moving downwards toward the lower-quality measurements of the sensors in the hierarchy. This means that the measurement of each sensor in the hierarchy below the top two sensors in the hierarchy only provides an input value to the state estimator that has not already been measured with a higher priority (in particular, higher quality) by one of the sensors above in the hierarchy.

[0052] According to a further advantageous embodiment, the system model and the respective measurement model and the mask determined in advance by means of the respective null space are stored on the computing unit.

[0053] According to a further advantageous embodiment, the computing unit is designed to calculate the respective null space of the measurement model before starting to generate a time series with estimates of at least one component of the system state. The system model and the respective measurement model can be already stored on the computing unit or can be newly added to it, for example, in the event of a sensor change or a change in the observed system.

[0054] A further aspect of the invention relates to a method for estimating at least one component of a system state of an observed system, wherein a computing unit receives data streams with measured values ​​from at least two different sensors and uses the data streams of the sensors as input variables of a state estimator implemented in the computing unit to determine a current estimate of the component of the system state, wherein a mathematical model of the observed system is implemented in the computing unit as a system model and a respective further mathematical model of the respective sensor is implemented as a respective measurement model, characterized in that a predefined hierarchy of the sensors is stored in the computing unit according to a predefined priority of their measurements, and the measurement of at least one of the sensors other than the one at the top of the hierarchy is masked by the computing unit with a mask,which is determined by the null space of the measurement model of at least the sensor immediately above in the hierarchy, so that measurements of at least two sensors are complementarily included in the state estimator, whereby the mask is a projection into the null space of the measurement model of at least the sensor immediately above in the hierarchy.

[0055] According to a further advantageous embodiment, the mask is a projection into the null space of the measurement model of at least the sensor immediately above it in the hierarchy.

[0056] According to a further advantageous embodiment, the mask is a projection onto the core of the measurement matrix of the measurement model of at least the sensor immediately above it in the hierarchy.

[0057] According to a further advantageous embodiment, a Kalman filter or a derivative of a Kalman filter is used as the state estimator.

[0058] According to a further advantageous embodiment, the measurement of at least one sensor other than the one at the top of the hierarchy is masked by the computing unit with a cumulative mask by determining the cumulative mask from each null space of the respective measurement model of all sensors located above it in the hierarchy, so that only those measurements of a sensor are included in the state estimator which are not made by any sensor higher in the hierarchy.

[0059] According to a further advantageous embodiment, the system model and the respective measurement model and the mask determined in advance by means of the respective null space are stored on the computing unit.

[0060] According to a further advantageous embodiment, the respective null space of the measurement model is calculated by the computing unit before the start of the generation of a time series with estimates of the at least one component of the system state.

[0061] Advantages and preferred developments of the proposed method result from an analogous and analogous transfer of the statements made above in connection with the proposed computing unit.

[0062] Further advantages, features, and details will become apparent from the following description, which—if appropriate, with reference to the drawings—describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0063] They show: Fig. 1: An exemplary system whose state is at least partially estimated by the computing unit according to an embodiment of the invention. Fig. 2: A vehicle as a considered system, the state of which is at least partially estimated by the computing unit according to an embodiment of the invention. Fig. 3: A mobile robot as a considered system, whose state is at least partially estimated by the computing unit according to an embodiment of the invention. Fig. 4: A robot arm as a considered system, the state of which is at least partially estimated by the computing unit according to an embodiment of the invention.

[0064] Fig. 1 shows a system 2 comprising a movable object, for the observation of which a first sensor 3, a second sensor 4, and a third sensor 5 are used. The data streams of the measurements from the sensors 3, 4, 5 are made available to the computing unit 1 for estimating at least one component of a system state of the observed system 2. The computing unit 1 uses the data streams from the sensors 3, 4, 5 as input variables of a state estimator implemented in the computing unit 1 to determine a current estimate of the component of the system state, for which purpose a mathematical model of the observed system 2 is implemented in the computing unit 1 as the system model and a respective further mathematical model of the respective sensor 3, 4, 5 is implemented as the respective measurement model. In the computing unit 1, a manually predefined hierarchy of the sensors 3, 4, 5 is also stored according to a predefined priority of their measurements. Fig. Figure 1 shows an example application of the state estimator: The task of the state estimator is to determine the position [x,y,z,] and the temperature θ of a point-like object moving in three-dimensional space. Therefore, for a vector system state, the following applies: xT=[x,y,z,θ]T

[0065] The computing unit 1 masks the measurement of at least one of the sensors 3, 4, 5 except for the one at the top in the hierarchy with a mask determined by the null space of the measurement model of at least the sensor 3, 4, 5 immediately above it in the hierarchy, so that measurements from at least two sensors 3, 4, 5 are included in the state estimator in a complementary manner. This happens as follows, since the different sensor systems have different principles and properties: The first of the sensors 3 provides precise measurements of the position of the object in system 2, but only indirectly by observing a shadow of the object on a flat surface, i.e. for this first sensor 3 only the projection onto a plane of the position is visible.

[0066] However, because of its accuracy, this first sensor 3 is defined as the primary sensor with its measurement 0 y ∈ ℝ 2 and 0H=[−101013−23130].

[0067] Another data source is the second sensor 4, which can directly measure the complete position of the object in space, therefore for this measurement 1 y ∈ ℝ 3 However, this measurement is subject to bias, ie there is a non-calibrated second sensor 4. The corresponding measurement matrix is ​​thus 1H=[100001000010].

[0068] Finally, there is the possibility of temperature detection by the third sensor 5, which allows a direct measurement 2 y ∈ ℝ 1 with 2 H = [0, 0, 0, 1]. The corresponding null-space projectors are: 0Pnull=13[1110111011100003],1Pnull=diag(0,0,0,1),2Pnull=diag(1,1,1,0).

[0069] The corresponding masks are: 1Φ=13[111111111],2Φ=1

[0070] It is evident that 1 Φ has too low a rank. With [U1TU2T]y=[S000][V1TV2T]x However, it is possible to define the minimal representation projector: S1ΦV1T,1Φ=[−13−13−13].

[0071] Analogously, the minimal representation projector for temperature measurement is trivial, since 2 Φ = 1 has full rank, therefore: S2ΦV1T,2Φ=1. Finally, the modified measurement matrix is: H˜=[−101013−23130−13−13−1300001]

[0072] Fig. Figure 2 shows another system 2 in the vicinity of a vehicle (passenger car). Robust and accurate perception of the environment is of great importance in the automotive industry when developing driver assistance systems. The detection and tracking of other road users and the environment in the sense of a considered system 2 is one of the core components for the further development of driver assistance systems up to and including autonomous driving. So-called "target tracking" attempts to estimate the position, speed, and relative orientation of objects 2 in traffic. Such objects 2 can be, among other things, other vehicles, pedestrians, or even static infrastructure. The sensors used are primarily cameras 3, RADARs 4 (specifically tailored to the automotive industry), and LiDARs 5, although the latter are not yet established in the mass market due to high costs.Sensors 3, 4, and 5 complement each other: LiDARs 4 and RADARs 5 measure the environment in the form of 3D point clouds, which can, however, be ambiguous and noisy. Cameras 3 provide clear measurements with little noise, but are limited to two dimensions and can also be severely limited by external lighting conditions. However, a computing unit 1 can combine all sensor types of sensors 3, 4, and 5 by applying supplementary state estimation: The two-dimensional camera information from cameras 3 is used as the basis and supplemented with the RADAR / LiDAR information only in the dimension outside the image plane. This is achieved by projecting the measurements from RADAR 4 and LiDAR 5 into a subspace, the null space of the camera measurement matrix.

[0073] Fig. Figure 3 shows a mobile robot 2 as the observed system, which is navigating on uneven terrain. Thus, the entire six-dimensional pose (rotation and position) of the robot 2 must be determined. For this task, the robot 2 has an inertial measurement unit (IMU) 3 as the first sensor, which provides the gravitational vector, from which the pitch and roll angles of the robot 2 can be determined – however, this first sensor 3 suffers from strong measurement noise. In addition, there are two GNSS antennas 4, 5, which are mounted at a distance from each other. Together with a base station, highly accurate real-time kinematics (RTK) position data can be generated. The two GNSS antennas 4, 5 can be used to determine the three-dimensional position of the robot 2 with high precision, as well as its yaw and roll angles, but not the pitch angle – thus a total of five of the six pose dimensions.This is achieved by combining the two measured antenna positions with the known distance between the two antennas. See https: / / doi.org / 10.1002 / rob.22016, Section 4.3. The complementary filtering approach allows the measurement of IMU 3 to be masked so that it only contributes the pitch angle to the state estimation, while all other five measurement dimensions are covered by the RTK-GNSS system 4, 5. The mask used by computing unit 1 is a projection matrix into the null space of the GNSS measurements, which is also applied to the IMU measurements.

[0074] Fig.Figure 4 shows an industrial robot manipulator as System 2 under consideration. The pose of the end effector (e.g., hand or gripper) of a robot arm is six-dimensional (position and orientation). Using two exemplary joint angle sensors 3, 4 and the known kinematic structure, the pose of the end effector at the distal end of the robot arm can be calculated. However, this so-called forward kinematics can be distorted by elastic bending in the mechanical structure or incorrect measured values ​​from the joint angle sensors 3, 4. Complementary to the forward kinematics, the position of the end effector is measured with high precision by a camera as the third sensor 5 – in two-dimensional image coordinates. Since the camera measurements, unlike the forward kinematics, are not negatively influenced by systematic errors, they should be used primarily.However, the camera information alone is not sufficient, as it only represents a two-dimensional measurement of the six-dimensional state. The complementary state estimation generates a mask (four-dimensional) from the null space of the camera measurement for each measurement step, thereby masking the forward kinematics and generating a complementary measurement of the end-effector pose in six dimensions, with two coming from camera 5 and four from the forward kinematics, calculated using known robot geometry and the joint angle sensors 3, 4.

[0075] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that exemplary embodiments are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description. List of reference symbols 1 computing unit 2 observed system 3 first sensor 4 second sensor 5 third sensor

Claims

[1] A computing unit (1) for estimating at least one component of a system state of an observed system (2), wherein the computing unit (1) is designed to receive data streams from measurements of at least two different sensors (3, 4, 5) and to use the data streams of the sensors (3, 4, 5) or data products derived from sensor measurements as input variables of a state estimator implemented in the computing unit (1) to determine a current estimate of the component of the system state, wherein a mathematical model of the observed system (2) is implemented as a system model and a respective further mathematical model of the respective sensor (3, 4, 5) is implemented as a respective measurement model in the computing unit (1), characterized byin that a predefined hierarchy of sensors (3, 4, 5) is stored in the computing unit (1) according to a predefined priority of their measurements, and wherein the computing unit (1) is designed to mask the measurement of at least one of the sensors (3, 4, 5) other than the one at the top in the hierarchy with a mask which is determined by the null space of the measurement model of at least the sensor (3, 4, 5) immediately above it in the hierarchy, so that measurements of at least two sensors (3, 4, 5) are complementarily included in the state estimator, wherein the mask is a projection into the null space of the measurement model of at least the sensor (3, 4, 5) immediately above it in the hierarchy. [2] Computing unit (1) according to claim 1, wherein the mask is a projection onto the core of the measurement matrix of the measurement model of at least the sensor (3, 4, 5) immediately above in the hierarchy. [3] Computing unit (1) according to one of the preceding claims, wherein the state estimator is a Kalman filter or a derivative of a Kalman filter. [4] Computing unit (1) according to claim 3, wherein the computing unit (1) is designed to apply the respective mask used for the measurement to a covariance matrix of the Kalman filter or the derivative. [5] Computing unit (1) according to one of the preceding claims, wherein the computing unit (1) is designed to mask the measurement of at least one sensor (3, 4, 5) other than the one at the top in the hierarchy with a cumulative mask by determining the cumulative mask from each null space of the respective measurement model of all sensors (3, 4, 5) located above it in the hierarchy, so that only those measurements of a sensor (3, 4, 5) are included in the state estimator which are not made by any sensor (3, 4, 5) located higher in the hierarchy. [6] Computing unit (1) according to one of claims 1 to 5, wherein the system model and the respective measurement model and the mask determined in advance by means of the respective null space are stored on the computing unit (1). [7] Computing unit (1) according to one of claims 1 to 5, wherein the computing unit (1) is designed to calculate the respective null space of the measurement model before starting to generate a time series with estimates of the at least one component of the system state. [8] Method (1) for estimating at least one component of a system state of an observed system (2), wherein a computing unit (1) receives data streams from measurements of at least two different sensors (3, 4, 5) and the data streams of the sensors (3, 4, 5) are used as input variables of a state estimator implemented in the computing unit (1) to determine a current estimate of the component of the system state, wherein a mathematical model of the observed system (2) is implemented as a system model and a respective further mathematical model of the respective sensor (3, 4, 5) is implemented as a respective measurement model in the computing unit (1), characterized bythat a predefined hierarchy of sensors (3, 4, 5) is stored in the computing unit (1) according to a predefined priority of their measurements, and the computing unit (1) masks the measurement of at least one of the sensors (3, 4, 5) other than the one at the top in the hierarchy with a mask which is determined by the null space of the measurement model of at least the sensor (3, 4, 5) immediately above it in the hierarchy, so that measurements of at least two sensors (3, 4, 5) are complementarily included in the state estimator, the mask being a projection into the null space of the measurement model of at least the sensor (3, 4, 5) immediately above it in the hierarchy.

Images