Method for determining suitability evaluation data set of virtual sensor calculation function

Abstract

The invention relates to a method for evaluating a data set with respect to the suitability for determining a calculation function of a virtual sensor, comprising the following steps:-providing the data set, the data set comprising measurement data, the measurement data being generated from measurements of measurement variables by at least two real sensors, the measurement data being generated from the measurement variables by at least two real sensors; the measurement data has a respective dimension of a respective real sensor of the at least two real sensors; providing an input region, the input region being defined for the measured variables of the at least two real sensors in order to specify at least one requirement for determining a calculation function; determining, using a machine learning model, a coverage between the data set and the provided input area; and evaluating the data set based on the determined coverage. The invention also relates to a computer program, a device and a storage medium for this purpose.

Description

Technical Field

[0001] The invention relates to a method for evaluating a data set with respect to its suitability for determining a calculation function of a virtual sensor. The invention also relates to a computer program, a device and a storage medium for this purpose. Background Art

[0002] The selection of a suitable dataset for a virtual sensor may be difficult for a number of reasons. First, the effectiveness of a virtual sensor depends inter alia on the quality and relevance of the data on which it is based. Data that is inaccurate, incomplete or does not represent the expected conditions of use may lead to faulty or inaccurate predictions. Second, differences in operating conditions or in context between the data source and the planned use of the virtual sensor may limit the applicability of the data. Finally, technical aspects, such as the need to integrate data from different sources or to process and store large datasets, may also make the selection of a suitable dataset difficult.

[0003] In the prior art, when designing an experiment, a data set is collected, for example using prior knowledge or also using classical Design of Experiment (DoE) methods. Known DoE methods are, for example, response surface methods based on simple approximation models such as linear models or splines. DoE methods focus in particular on how a function or a virtual sensor changes within an input region. In addition, various test methods are known, for example, for analyzing whether there is enough data in a sub-region.

[0004] However, in particular for nonlinear virtual sensors, it is difficult to determine a sufficient amount of data. For example, it may also be difficult to establish the security of the virtual sensor.

[0005] DE 102021212737A describes a computer-implemented method for checking test and / or training data sets for a computer-based machine learning module. DE 69800186 T2 describes virtual vehicle sensors based on neural networks that are trained with the aid of data generated by a simulation model. A professional overview of the topics of machine learning and data mining is given in the chapter "Machine Learning and Data Mining" of Wolfgang Ertel's book "Basic Course in Artificial Intelligence: A Practice-Oriented Introduction". Summary of the invention

[0006] The subject matter of the invention is a method having the features of claim 1, a computer program having the features of claim 7, a device having the features of claim 8 and a computer-readable storage medium having the features of claim 9. Further features and details of the invention are derived from the respective dependent claims, the description and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program according to the invention, the device according to the invention and the computer-readable storage medium according to the invention, and vice versa, respectively, so that the disclosure of the individual inventive aspects always refers to one another or can refer to one another.

[0007] The subject matter of the invention is in particular a method for evaluating a data set with respect to its suitability for determining a calculation function of a virtual sensor, the method comprising the following steps:

[0008] - providing a data set, wherein the data set comprises measurement data, wherein the measurement data are generated from a measurement of a measurement variable by at least two real sensors, wherein the measurement data have a respective dimension of a respective real sensor of the at least two real sensors;

[0009] - providing an input area, wherein the input area is defined for the measurement variables of the at least two real sensors in order to specify at least one requirement for determining the calculation function;

[0010] - in the case of a machine learning model, determining the coverage between the dataset and the provided input region;

[0011] - Evaluating the dataset based on the determined coverage.

[0012] The calculation function of the virtual sensor can in particular calculate another measured variable based on the measurement data of the at least two real sensors. This other measured variable may be difficult to measure or completely impossible to measure directly, for example, and therefore, this other measured variable can be indirectly determined by the virtual sensor via further measurements of the real sensor. It is also conceivable to stagger the arrangement so that these measurement data are generated from the measurements of other virtual sensors, which in turn can be based on the measurements of the real sensors. Alternatively, for example, the real sensor and the virtual sensor may also provide a data set for the calculation function of the virtual sensor through corresponding measurements. Therefore, the virtual sensor is, for example, a software function, with which a parameter that cannot be measured or is difficult to measure can be derived from other measurable or available sensor data and information. The requirement for determining the calculation function can be, for example, that the corresponding real sensor is within the area allowed by the sensor. Thus, for example, it is conceivable that for a temperature sensor, the following area is given by the physical properties of the temperature sensor, within which the temperature sensor outputs a reliable measurement result. In other words, the coverage range particularly describes the proportion of the data set covering the input area. The input area can be the following area, in which the data points of the data set can theoretically appear, or can be measured or determined. In one example, the machine learning model can additionally determine the calculation function of the virtual sensor. Alternatively, the calculation function of the virtual sensor can be determined, for example, by another machine learning model. Based on the coverage, the quality of the data set can be described with respect to its suitability for determining the calculation function of the virtual sensor. Based on this, the data set or future data sets can then be advantageously improved, and the virtual sensor can also be improved thereby, whereby, for example, the risks when using virtual sensors, for example when using virtual sensors in autonomous systems, can be reduced.

[0013] Within the scope of the present invention, it is also conceivable that the input area is defined by the range between the corresponding minimum and maximum values ​​of each of the at least two real sensors, wherein the range between the corresponding minimum and maximum values ​​represents the reliable range of the corresponding real sensor. In other words, the reliable range refers in particular to the range for which the real sensor is designed. This may be due to the physical properties of the real sensor. The range between the minimum and maximum values ​​can be predetermined by the specifications of the corresponding real sensor. By defining the range for each of these real sensors, it is advantageous to only consider the measurement data in the reliable range from the corresponding real sensor and use it for the calculation function of the virtual sensor. In addition, the input area can be defined taking into account the knowledge of the physical background or characteristics of the common range of the corresponding real sensor. Alternatively, it is also conceivable that the input area is defined as at least one sub-range between the corresponding minimum and maximum values ​​of each of the at least two real sensors in order to determine the coverage within the at least one sub-range. The range between the corresponding minimum and maximum values ​​of each of the at least two real sensors can be divided into at least two sub-ranges, for example, in order to determine the coverage within the at least two sub-ranges. Thereby, a comparison between the coverage rates of the at least two sub-ranges can be advantageously achieved.

[0014] It is also feasible that the determination of the coverage rate further comprises the following steps:

[0015] - calculating an extension area for each data point of the data set in order to determine the coverage rate based on the calculated extension areas of the data points, wherein the extension area is specific to the area covered by the respective data point in the input area.

[0016] In other words, the extended area preferably represents an area for which, based on the knowledge extracted by the machine learning model, it can be considered that the area is covered by the corresponding data points.

[0017] Within the scope of the present invention, a further advantage can be achieved if the calculation of the extension area also comprises the following steps:

[0018] - Determine a next adjacent data point for each data point in the data set so as to calculate the expansion area based on a linear model, in particular a local linear model, using the data point and the next adjacent data point, wherein the maximum radius of the expansion area reaches at most the next adjacent data point of each data point.

[0019] In particular, the next adjacent data point is selected as the boundary because, starting from the next adjacent data point, the next adjacent data point itself provides better information about the coverage and, in addition, also provides better information about the calculation function of the virtual sensor. The linear model preferably describes a relationship, in particular a straight line, between the respective data point and the respective next adjacent data point.

[0020] In the present invention, it can be advantageously provided that the method further comprises the following steps:

[0021] -Select the corresponding data point;

[0022] - determining the next adjacent data point of the selected data point;

[0023] - in the case of a machine learning model, determining a local model for the selected data points;

[0024] - determining a linear model, wherein the linear model describes the relationship between the selected data point and the next adjacent data point;

[0025] - comparing the local model with the linear model in order to calculate the extended region based on deviations between the local model and the linear model.

[0026] The local model can be approximated based on the corresponding values ​​of the weights of the machine learning model for the corresponding data points and can preferably be a local linear model. In other words, the expansion area is especially derived from the machine learning model. The machine learning model preferably represents the knowledge extracted from the data. Based on the machine learning model, a local linearity assumption can be assumed. The assumption is especially checked and, depending on the realization of the assumption, the expansion area can be determined.

[0027] Another advantage within the scope of the present invention can be achieved if the method further comprises the following steps:

[0028] - extending the data set with further data points in order to increase the coverage of the data set by the further data points, wherein the further data points are located in areas of the input area that are not covered by the data set.

[0029] The expansion of the data set may be advantageous in improving the quality of the data set for determining the computational function of the virtual sensor. Thus, a higher coverage may advantageously contribute to the virtual sensor providing a more reliable output. The expansion of the further data points in the area not covered by the data set may be performed in various ways, such as randomly or by determining the distances between the respective data points and providing respective new data points in the middle of the respective determined distances. The expansion of the data set with further data points is preferably performed until the coverage exceeds a defined threshold.

[0030] It is also conceivable that the calculation function of the virtual sensor is applied to calculate the injection quantity of the injector of the engine, wherein the method comprises the following steps:

[0031] - providing the data set, wherein the data set comprises measurement data, wherein the measurement data are generated from a measurement of a measurement variable by means of the at least two real sensors, wherein the at least two real sensors measure at least pressure and / or temperature, wherein the measurement data have a respective dimension of a respective real sensor of the at least two real sensors;

[0032] - providing the input area, wherein the input area is defined for the measurement variable in order to specify at least one requirement for determining the calculation function, wherein the input area represents an area having at least two dimensions for which the at least two real sensors are specified;

[0033] - determining a coverage ratio between the data set and the provided input region using a neural network, wherein the neural network comprises a linear activation function, wherein the coverage ratio represents the extent to which the input region is covered by the data set;

[0034] - Evaluating the dataset based on the determined coverage.

[0035] The calculated injection quantity of the injector can advantageously be used to calibrate the injectors of the engine.

[0036] The method according to the invention can be used in a vehicle. The vehicle can be designed, for example, as a motor vehicle and / or a passenger vehicle and / or an autonomous vehicle. The vehicle can have a vehicle device, for example, for providing an autonomous driving function and / or a driver assistance system. The vehicle device can be designed to at least partially automatically control and / or accelerate and / or brake and / or steer the vehicle.

[0037] The subject of the invention is also a computer program, in particular a computer program product, which includes instructions which, when executed by a computer, cause the computer to carry out the method according to the invention. The advantages brought about by the computer program according to the invention are thus the same as those already described in detail with respect to the method according to the invention.

[0038] The subject matter of the invention is also a data processing device which is set up to carry out the method according to the invention. For example, a computer can be provided as the device, which executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program can be stored and from which the processor can read the computer program for execution.

[0039] The subject of the invention may also be a computer-readable storage medium having a computer program according to the invention and / or comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the invention. The storage medium is designed, for example, as a data memory, such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium may be integrated, for example, into the computer.

[0040] Furthermore, the method according to the invention can also be implemented as a computer-implemented method. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Other advantages, features and details of the present invention are apparent from the following description, in which embodiments of the present invention are described in detail with reference to the accompanying drawings. The features mentioned in the claims and in the description may reflect the essence of the invention individually or in any combination.

[0042] in:

[0043] Figure 1 Schematic visualizations of methods, apparatus, storage media, and computer programs according to embodiments of the present invention are shown;

[0044] Figure 2 A schematic diagram showing an input area, a data point and an expansion area according to an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of an engine, a virtual sensor and a real sensor according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0046] exist Figure 1 , a method 100, an apparatus 10, a storage medium 15 and a computer program 20 according to an embodiment of the present invention are schematically shown.

[0047] Figure 1A method 100 for evaluating a data set with respect to suitability for determining a calculation function of a virtual sensor according to an embodiment of the present invention is shown. In a first step 101, a data set is preferably provided, wherein the data set comprises measurement data, wherein the measurement data are generated from the measurement of a measurement variable by at least two real sensors, wherein the measurement data have a corresponding dimension of a corresponding real sensor of the at least two real sensors. In a second step 102, an input region is preferably provided, wherein the input region is defined for the measurement variables of the at least two real sensors in order to specify at least one requirement for determining the calculation function. In a third step 103, a coverage ratio between the data set and the provided input region can be determined using a machine learning model. In a fourth step 104, the data set can be evaluated based on the determined coverage ratio.

[0048] Figure 2 An input region 3 is schematically shown, within which a plurality of data points 4 of the data set are located. According to an embodiment of the present invention, in order to determine the coverage rate, an extension region 5 around each of these data points 4 may be calculated.

[0049] Figure 3 An engine 7 is schematically shown, comprising an injector 6 , a virtual sensor 1 and two real sensors 2 .

[0050] In particular, the coverage describes a measure that allows conclusions to be drawn as to whether there are sufficient data points 4 in the data set that are relevant to the function to be learned of the virtual sensor 1. This can be performed using a neural network.

[0051] Preferably, the approximation model is adapted to the variability of the data and a measure is derived therefrom which indicates how well these data can characterize the variability of the virtual sensor 1 in the output space. The data quality is preferably determined iteratively. Compared to classical design of experiments (DoE) methods, more complex function models can be mapped by using neural networks.

[0052] The invention can be used to analyze data obtained from a real sensor 2. The real sensor 2 can determine measurements of the environment in the form of sensor signals, which can be given, for example, by specific data, i.e. sensor values ​​of a system from which unmeasurable quantities are to be derived, such as the injection mass derived from quantities characterizing pressure or other physical quantities associated with the operation of the injector.

[0053] In order to provide a virtual sensor 1 , information about an element encoded by the sensor signal may be obtained based on the sensor signal, ie an indirect measurement may be performed based on the sensor signal serving as a direct measurement.

[0054] The invention can also be used to calculate control signals for controlling a technical system, such as a computer-controlled machine, a robotic system, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system. The virtual sensor 1 is used in particular to control and regulate the system, for example to (better) regulate the injection process.

[0055] The method of the present invention may be used to generate test data, verification data and / or validation data in order to check whether a trained ML system can be safely operated.

[0056] The quality metric according to an embodiment of the invention allows in particular to derive data areas in which data points 4 are missing. Further data points 4 can then be collected locally in this area. In this case, the existing data points 4 are preferably used as prototypes for generating further data points 4. Furthermore, it may be important not only to achieve coverage, but also to specify which areas are able to achieve this coverage.

[0057] In a first step, a machine learning model can be trained based on a DoE model, in particular a response surface. Based on the network and the coverage, new data points 4 are preferably generated. The step of generating data points is preferably performed until the coverage of the data set exceeds a defined threshold. The threshold of coverage can vary within the input region.

[0058] From this coverage, the quality of the data set can be seen. This can apply to training data, validation data and test data. Based on this, the data set can then be advantageously improved, and thereby the virtual sensor 1 can also be improved or improvements related to the test data set can be made and the risks when using the virtual sensor 1 in an autonomous system can be reduced.

[0059] In the following, inter alia an incremental method for determining the coverage is described. The method is preferably based on a principal component analysis (PCA) based dimensionality reduction and a kd-tree based partitioning of the input region.

[0060] Figure 2 An overview of possible implementations of the method is given.Based on the evaluation of the prediction quality for the data points 4, it is determined how each data point 4 can be locally generalized under a certain hypothesis, which is presented with an expansion region 5.

[0061] The idea behind the method is in particular to consider regression tasks and the corresponding requirements for error limits. Since virtual sensors 1 are preferably considered, the task can be a regression task and the machine learning model can have the form y=f(x), where and Preferably, it is a vector of physical quantities. The ground truth label associated with point x can be represented by a label function g. In other words, the label function g preferably represents the true value of the sensor, which is derived from x.

[0062] First, we can formulate the absolute accuracy requirement and error limit of the prediction: e(x) = |g(x)-f(x)|≤∈. The goal can be: find a k Around x k Sphere B, that is, B rk , where x k is the corresponding data point 4. The sphere B(x k ) in particular determines the part of the input area that is considered to be covered. This is in particular Figure 2 The circle around the data point 4 in , or the extended area 5. k Some points to consider are as follows. If the prediction error for this data point is too large: (e(x k )>∈), then the sphere can be degenerated into r k = 0. If (e(x k )<∈), then e(x k ) is smaller, the radius r k The larger the detection range, the larger the detection range, especially not just the local point x k is determined by, and preferably primarily by, the underlying function g(x) being true and, correspondingly, e(x) at x k It is determined by how the environment behaves.

[0063] In particular, neither g(x) nor f(x) is known. Thus, preferably assumptions are made in order to determine the coverage. To illustrate, an intuitive approach based on a local linear model may be chosen. Since the machine learning model preferably models similar models, the assumptions may advantageously fit the model class of the machine learning model. In particular, for the representation of g(x), the nearest data point 4, i.e., the nearest neighbor x, may be chosen. NN,k , as in x k The nearest sample of g(x) in the neighborhood of is used for its local representation. k ) can be limited to the distance to the nearest neighbor, i.e. In addition, f(x) can be locally approximated as a linear model The linear model is represented by x k Similarly, g(x) can be implied locally based on the (activation-adjusted) weights in x. k and x NN,k The values ​​at are linearly interpolated, which is particularly referred to within the scope of the present invention as Next, for example, we use two linear approximations and calculate

[0064] The motivation behind this approach could be counterfactual hypothesis analysis, where the local model is determined Is it sufficient to detect x k and x NN,k or determine whether a change in the model is necessary. and The point whose difference is greater than ∈ can determine the radius of the sphere. k ) is still within the ∈ limit, but within the scope of the present invention, especially using To determine the radius of the covering sphere, i.e. the local linear approximation prediction.

[0065] Based on this formula, the covering sphere B(x k )’s radius r k Determined as r k =s*|x k -x NN,k |, where s∈[0,1], where

[0066]

[0067] The entire input region 3 can be trained on , because in particular one should not usually risk unreliable predictions outside this region. To achieve coverage of this input region, one may need: for a set of data points X, the union of the spheres surrounding the data points 4 of the training region covers Or intuitively, Figure 2 If T is covered by the data point coverage of the dataset, then the dataset can be considered to be densely sampled enough from a coverage perspective under the above assumptions, or the extent to which the input region 3 is covered can be determined.

[0068] The above description applies in particular to a sphere, ie a uniform extension in different dimensions with respect to the measured variable. However, the method can also be easily extended with physical a priori knowledge of the different extensions in different dimensions.

[0069] Preferably, a kd-tree approach is used, i.e. in particular a Hyperbox, in order to provide a dataset having all the advantages, such as simplicity and interpretability, and all the disadvantages, such as proximity to boundary points. In a possible experiment, some of the above models with different dimensions can be selected and their coverage compared. In particular, (i) the coverage in various sub-regions of the input region given by the kd-tree and (ii) the average coverage of these regions can be analyzed. Furthermore, the percentage of the input region covered can be approximated by Monte-Carlo sampling, i.e. samples can be drawn from T and it can be checked how many points are in Inside.

[0070] Machine learning models with smaller input dimensions can have better coverage than larger machine learning models. It can also be found that the coverage within an input region can differ significantly between smaller machine learning models and larger machine learning models. This can be for two reasons: (i) the predictions may have large errors, or (ii) the machine learning models may vary significantly in a certain portion of the input space. Furthermore, machine learning models of different sizes may perform better or worse within a particular region of data consistently. This can mean that even in a variety of situations, critical data regions can be identified that degrade coverage and thereby increase assessment risk, and may impact machine learning model performance due to lack of training data.

[0071] The above description of the embodiments only describes the present invention within the framework of examples. Of course, as long as it is technically reasonable, the various features of these embodiments can be freely combined with each other without departing from the protection scope of the present invention.

Claims

1. A computer-implemented method (100) for evaluating a data set with respect to suitability for determining a calculation function of a virtual sensor (1), the method comprising the following steps: - providing (101) the data set, wherein the data set comprises measurement data, wherein the measurement data are generated from the measurement of a measurement variable by at least two real sensors (2), wherein the measurement data have a respective dimension of a respective real sensor of the at least two real sensors (2); - providing (102) an input area (3), wherein the input area (3) is defined for the measurement variables of the at least two real sensors (2) in order to specify at least one requirement for determining the calculation function; - determining (103) the coverage between the data set and the provided input area (3) using a machine learning model; - evaluating (104) the data set based on the determined coverage, The method (100) further comprises the following steps: - extending the data set by further data points (4) in order to increase the coverage of the data set by the further data points (4), wherein the further data points (4) are located in an area of ​​the input area (3) not covered by the data set.

2. The computer-implemented method (100) of claim 1, It is characterized in that The input region (3) is defined by a range between a corresponding minimum value and a corresponding maximum value of each of the at least two real sensors (2), wherein the range between the corresponding minimum value and the corresponding maximum value represents a reliable range of the corresponding real sensor (2).

3. The computer-implemented method (100) according to any one of the preceding claims, It is characterized in that The determination of the coverage rate also includes the following steps: - calculating an extension area (5) for each data point (4) of the data set so as to determine the coverage rate based on the calculated extension area (5) of the data point (4), wherein the extension area (5) is specific to the area covered by the corresponding data point (4) in the input area (3).

4. The computer-implemented method (100) of claim 3, It is characterized in that The calculation of the extended area (5) further comprises the following steps: - Determine the next adjacent data point for each data point (4) of the data set so as to calculate the expansion area (5) based on a linear model using the data point (4) and the next adjacent data point, wherein the maximum radius of the expansion area (5) reaches the next adjacent data point of each data point (4) at most.

5. The computer-implemented method (100) of claim 3, It is characterized in that The calculation of the extended area (5) further comprises the following steps: - Select the corresponding data point (4); - determining the next adjacent data point of the selected data point (4); - determining a local model of the selected data points (4) using the machine learning model; - determining a linear model, wherein the linear model describes the relationship between the selected data point (4) and the next adjacent data point; - comparing the local model with the linear model in order to calculate the extension area based on the deviation between the local model and the linear model (5).

6. The computer-implemented method (100) according to any one of the preceding claims, It is characterized in that The calculation function of the virtual sensor (1) is applied to calculate the injection quantity of an injector (6) of an engine (7), wherein the method (100) comprises the following steps: - providing the data set, wherein the data set comprises measurement data, wherein the measurement data are generated from the measurement of a measurement variable by means of the at least two real sensors (2), wherein the at least two real sensors (2) measure at least pressure and / or temperature, wherein the measurement data have a respective dimension of a respective real sensor of the at least two real sensors (2); - providing the input area (3), wherein the input area (3) is defined for the measurement variable in order to specify at least one requirement for determining the calculation function, wherein the input area (3) represents an area having at least two dimensions, for which the at least two real sensors (2) are specified; - determining a coverage ratio between the data set and a provided input region (3) using a neural network, wherein the neural network comprises a linear activation function, wherein the coverage ratio indicates the extent to which the input region (3) is covered by the data set; - evaluating the data set based on the determined coverage.

7. A computer program (20) comprising instructions which, when executed by a computer (10), cause the computer to perform the computer-implemented method (100) according to any one of the preceding claims.

8. A data processing device (10) configured to carry out the computer-implemented method (100) according to any one of claims 1 to 6.

9. A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer to perform the steps of the computer-implemented method (100) according to any one of claims 1 to 6.

Images