Method and apparatus for evaluating and selecting selection function for active learning of model

Abstract

The invention relates to a method for evaluating a selection function of an active learning method for training a data-based model, comprising the following steps: providing (S1, S2; s11, S12) a training data set and a verification data set, the training data set and the verification data set respectively assigning the input data points to the marks; -performing (S3) a training of the data-based model (21) on the basis of the training data set by means of an active learning method based on the selection function; generating a plurality of evaluation sets of the test data set by resampling from the verification data set; -determining (S7) a model quality and a measure of uncertainty of the model quality based on the generated test data set, based on a statistical evaluation of model performance of the data-based model; -based on the model quality and the uncertainty measure, reserving (S9) or discarding the selection function.

Description

Technical Field

[0001] The present invention relates to data-based models, such as neural networks, and in particular to methods for providing and improving the training process of data-based models. Background Art

[0002] Data-based models, such as neural networks and the like, outperform conventional mathematical or physically driven models for a variety of complex problems. However, the model quality of data-based models depends significantly on the quality and quantity of the training data sets used to train the data-based models. The labeling of input data points, i.e. the assignment of desired outputs to input data points, is time-consuming and / or costly and is therefore a cost-driving factor. Therefore, the effort of providing a sufficient number of training data sets can be reduced by, on the one hand, reducing the number of training data sets required to adequately train the data-based model and / or, on the other hand, not labeling redundant input data points. Summary of the invention

[0003] According to the invention, a method for evaluating a selection function according to claim 1, a method for training a data-based model according to the independent parallel claim and a corresponding device according to the independent parallel claim are provided.

[0004] Further refinements are described in the dependent claims.

[0005] According to a first aspect, a method for evaluating a selection function for an active learning method for training a data-based model is provided, the method comprising the following steps:

[0006] - Provide training and validation datasets that assign input data points to labels, respectively;

[0007] - performing training of a data-based model based on these training data sets by means of an active learning method based on the selection function;

[0008] - Generate multiple evaluation sets of the test dataset by resampling these validation datasets;

[0009] - determining a model quality and an uncertainty measure of the model quality based on a statistical evaluation of the model performance of the data-based model based on the generated test data set;

[0010] - Based on the model quality and the uncertainty measure, keep or discard the selected function.

[0011] The system functions of a technical system can be provided in whole or in part with the aid of a data-based model. Such a data-based model needs to be trained. In order to train a data-based model, training data sets are provided which describe the system behavior and assign model outputs to input data points.

[0012] The training of data-based models can be efficiently performed with the aid of active learning methods. Active learning is a promising strategy for training data-based models and a method for reducing the workload of labeling input data points by minimizing the number of training data sets required for efficient training. Here, one or more unlabeled input data points are iteratively selected or determined, for example from a specified number of input data points, with the aid of a selection function (acquisition function). The selection function is defined such that the input data points are selected so that the maximum improvement in the model quality of the data-based model, i.e. the model accuracy of the data-based model, is achieved when the training data set supplemented by the one or more selected labeled input data points is used for subsequent training. The possible selection functions are various and it is usually not easy to find out which selection function will lead to the fastest possible training of the data-based model, wherein the desired mapping quality corresponding to the specified task is achieved.

[0013] The validation data set has the same format as the training data set and is generated in a similar manner. The validation data set is only indirectly used for model training and is used to find out underfitting or overfitting of the training state of the model based on the data. The validation data set can be used in a manner known per se to assess the quality of the model and to check the interruption criteria of the training.

[0014] Based on the evaluation of the selection function in terms of the improvement of the model quality of the data-based model to be created by means of the active learning method, the selection function can be selected, which selects unlabeled input data points for subsequent label determination. Typically, a validation data set can be additionally used to evaluate the model quality, which is used to evaluate the selection function.

[0015] Conventionally, the selection function is evaluated by determining the model quality of the data-based model generated thereby using a set of validation data sets. However, if labeling the input data points to provide the validation data sets is laborious, this approach is uneconomical because it requires a larger number of validation data sets. Furthermore, the evaluation of the achieved model quality may be inaccurate because, in particular when the number of available validation data sets is small, the determined model quality is not representative due to a possibly non-representative distribution of the validation data sets.

[0016] By means of the above method, evaluation sets of the test data sets are generated from these validation data sets respectively by a resampling method, such as a bootstrapping method, so as to overcome the above problem when the number of validation data sets used to evaluate the quality of the implemented model is too small. In the resampling method, statistics are calculated based on multiple samples (evaluation sets) from these validation data sets.

[0017] Resampling is performed by randomly selecting test data sets multiple times (drag and drop) from the original set of validation data sets to generate multiple evaluation sets of the test data sets. The evaluation sets of test data sets thus obtained can have the same or different number of test data sets as validation data sets. Multiple evaluation sets allow to create statistics from the model quality, from which average model quality and uncertainty measures are generated, in particular in the form of confidence intervals. Model performance evaluates the statistical distribution of model quality.

[0018] By providing multiple evaluation sets of the test data set, an indication of model quality and an uncertainty measure of the model quality can be provided, which enables evaluation of the selection function for a training process using active learning. In particular, the indication of the model quality in combination with the corresponding uncertainty measure enables ascertaining how the use of the determined selection function affects the improvement of the model based on the data.

[0019] With the method described above, selection functions can be evaluated in an improved manner, according to which new input data points are selected in active learning for the subsequent determination of labels (annotations). In this case, the model quality and uncertainty measures, i.e. the corresponding confidence intervals, are calculated with the aid of resampling or bootstrapping methods, whereby the model quality of the data-based model can be evaluated after training using an active learning method based on a specified selection function, and thus the suitability of the selection function can be evaluated.

[0020] In general, the above method enables identification of whether the selection function used for a particular model training of a data-based model is better or worse than training performed using another selection function, or whether this simply occurred by chance through a favorable choice of a test dataset.

[0021] Furthermore, the selection function used can be confirmed or rejected depending on the model quality and the uncertainty measure, in particular according to an importance measure that depends on the model quality and the uncertainty measure, in particular based on a threshold comparison.

[0022] The importance measure can be determined, for example, by means of a weighted sum of the model quality and the uncertainty measure.

[0023] In particular, the selection function used can be used to create one or more additional input data points or to select one or more additional input data points from the provided input data points within the framework of an active learning training process, wherein labels are determined for these additional input data points in order to generate further training data sets in this way, wherein these further training data sets are added to the existing training data set and further training of the data-based model is carried out.

[0024] Furthermore, the training of the data-based model can be performed multiple times based on these training data sets by means of an active learning method based on the selection function.

[0025] According to one embodiment, the method can be performed to train a first data-based model and a second data-based model by means of a first selection function and a second selection function different from the first selection function, wherein the obtained model quality and the corresponding confidence level are determined for the first selection function and the second selection function, respectively, wherein the first or second selection function is selected based on a comparison based on the obtained model quality and the corresponding uncertainty measure or based on the obtained importance measure in order to generate other training data sets and further train the data-based models accordingly. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Subsequently, the embodiments are described in more detail with reference to the accompanying drawings, wherein:

[0027] Figure 1 A schematic diagram of a technical system having actuators and a control unit is shown, in which a data-based model is implemented in order to carry out regulation or control of the actuators of the technical system;

[0028] Figure 2 A flow chart illustrating a method for training a data-based model by means of an active learning method and for evaluating a selection function is shown; and

[0029] Figure 3 A flow chart illustrating a method for training a data-based model and for evaluating and selecting one of two selection functions to perform further training is shown. DETAILED DESCRIPTION

[0030] Figure 1 The schematic diagram shows a technical system 1 which has a control unit 2 , an actuator 3 and a sensor system 4 .

[0031] The actuator 3 can have one or more actuators or the like in order to convert the control variable A into a physical influencing variable of the technical system 1 , such as a movement, a temperature change, an emission of electromagnetic radiation or the like of the technical system 1 or a part thereof.

[0032] The sensor device 4 can detect one or more system parameters, such as rotational speed, pressure, electrical parameters, temperature, position of components, etc., and / or one or more environmental parameters, such as ambient temperature, air pressure, etc., as sensor parameters with the help of one or more sensor units, and provide these sensor parameters to the control unit 2.

[0033] In the sense of control or regulation or execution of an algorithm, in particular based on sensor variables of the sensor device 4, the control unit 2 can provide one or more control variables A for controlling the actuator 3. The functions in the control unit 2 can be fully or partially performed with the aid of at least one data-based model 21. The data-based model 21 maps one or more sensor variables or variables derived therefrom and / or one or more default variables onto at least one system variable. The system variable can correspond directly or indirectly to the control signal A.

[0034] The data-based model 21 can correspond to a neural network, a Gaussian process or other trainable non-parametric function. The data-based model is trained in the following manner: one or more system parameters are assigned to one or more sensor parameters and / or one or more default parameters that form the input data points as a whole. The data-based model is trained based on training data sets, which respectively include input data points and one or more system parameters assigned to the input data points as labels. The labels of the corresponding input data points can usually be determined by complex measurements on a test bench or in the field or by simulation. Usually, especially in test bench measurements, the cost of determining the labels for training data-based models is quite large, and it is therefore desirable to reduce the number of training data sets required for training data-based models or keep the number as small as possible.

[0035] Examples of technical systems which use a data-based system model 32 and at least one signal time series in an input data set to be evaluated by the system model 32 are various.

[0036] For example, if a signal time series in the form of a motor current profile over a predetermined period of time in the past is determined as part of an input data set to be evaluated correspondingly by the data-based model 21, the rotational speed of the motor can be determined, if necessary, together with the additionally detected motor temperature, by evaluation in a system model trained in a corresponding manner. Other variables of the input data set can be the motor temperature and / or one or more variables determining the motor load. The signal time series indicates the profile of the motor current up to a specified current evaluation time point. The evaluation results in the current rotational speed of the motor as an output variable.

[0037] As another example, a data-based model 21 can be used to determine the amount of fuel injected in the injection system of an internal combustion engine. Here, the fuel for operating the internal combustion engine is injected into the combustion chamber of the cylinder of the internal combustion engine via an injection valve. To this end, the fuel is delivered to the injection valve via a fuel supply line, via which the fuel is provided at a high fuel pressure in a manner known per se (e.g., a common rail). The injection valve can have an actuator unit that can be controlled electromagnetically or piezoelectrically, which is coupled to a valve needle. By controlling the actuator unit, the valve needle moves in the longitudinal direction and releases a portion of the valve hole so that the pressurized fuel is injected into the combustion chamber of the cylinder. With the help of a piezoelectric sensor, the pressure change in the fuel guided through the injection valve can be determined as a voltage signal, the voltage signal is sampled and the voltage signal is provided as a sensor signal time series. The injection amount can be determined in a manner known per se by knowing the exact opening time point of the injection valve. The opening time is determined by a correspondingly trained data-based model 21 based on an input data set, which includes: a signal time series of a voltage signal as a vector; and a fuel pressure. The data-based model 21 can then output the opening time as a probability vector or directly as a time variable.

[0038] As another example, the data-based model 21 can be used to predict the driver state of a motor vehicle, such as for identifying driver fatigue. For this purpose, driving-related parameters can be provided as time curves in the form of signal time series, such as curves of steering position, curves of driving speed and / or curves of the driver's head posture. The output parameters after evaluation in the correspondingly trained system model result in the driver's excessive fatigue probability.

[0039] Figure 2 A flow chart illustrating a method for training a data-based model by means of an active learning method is shown.

[0040] First, in step S1 , one or more input data points are provided. As described above, these input data points are respectively composed of one or more sensor parameters and / or one or more default parameters.

[0041] To this end, in step S2, as described above, labels are determined from one or more system parameters, so that a training data set or a validation data set is formed from the input data points and the assigned labels. A set of multiple training data sets and a set of multiple validation data sets are obtained.

[0042] Now, in step S3, the data-based model is trained with the aid of one or more training data sets or supplementary training data sets as described below. The training is performed according to methods known per se, for example based on a back-propagation method.

[0043] In step S4, it is checked whether further training data sets are needed to improve the data-based model according to the specified interruption criteria. For example, a validation data set can be used to check the interruption criteria for model training. If this is the case (option: yes), the method is continued with step S5. Otherwise (option: no), the method is continued with step S7.

[0044] Then, in step S5, a set of a plurality of further input data points is provided as candidates for one or more training data sets to be determined subsequently. One or more input data points are selected from these input data points for labeling by means of a specified selection function.

[0045] In a subsequent step S6 , the corresponding one or more selected input data points are provided with a label on a test bench or by on-site measurement or by simulation, in order to form a new training data set and / or validation data set.

[0046] By returning to step S3, the data-based model is retrained or retrained using the training data set supplemented with the new training data set.

[0047] In step S7, the model quality of the trained data-based model 21 is evaluated. This evaluation is performed with the aid of evaluation sets of the test data set, which are generated from the existing validation data set with the aid of a resampling method.

[0048] The evaluation is performed by statistical evaluation of quality metrics generated for each test data set from the evaluation set of the test data sets. These quality metrics can be generated, for example, from the normalized mean deviation (L2 norm, etc.) between the labels of the test data set of the corresponding evaluation set and the model output of the data-based model for the corresponding input data points of the test data set of the corresponding evaluation set.

[0049] Model quality is derived as the mean or median of these quality measures and uncertainty measures / confidence are derived from the divergence or variance of these quality measures on the evaluation set with respect to the test dataset.

[0050] In step S8 , the model quality and its uncertainty measure may be evaluated according to an evaluation criterion.

[0051] The evaluation criterion may, for example, provide a threshold comparison with a specified importance measure threshold, wherein an importance measure may be formed which, for example, corresponds to a weighted sum of the model quality or the inverse of the model quality and the inverse of the uncertainty measure or the uncertainty measure.

[0052] If the evaluation criterion is not met (option: No), a new selection function can be provided in step S9 (e.g. from a list of specified selection functions) for retraining or further training of the data-based model according to the described method, and the method can be continued with step S4. Otherwise (option: Yes), the method can be terminated or the training of the data-based model can be continued in step S10 using the existing selection function by means of an active learning method.

[0053] The convergence of the above-described methods to active learning depends largely on the choice of the selection function. Currently, the selection function is determined empirically and should be done in such a way that with each newly added training data set, the improvement of the data-based model is optimized as much as possible. For this reason, in order to evaluate the model quality, it is proposed that a test data set is determined from the corresponding validation data set as the basis according to a resampling method, and the model quality and the uncertainty measure about the model quality are determined accordingly. Since the number of available evaluation sets of the test data set can be selected to be arbitrarily large by means of sampling according to the resampling method without the need for additional measurements, a statistical evaluation can now be performed, which can indicate the model quality and the uncertainty measure when evaluating the model quality. The method can be applied even when the number of available validation data sets is small.

[0054] Figure 3 A method for training a data-based model by means of two competing selection functions using active learning is explained with the aid of a flow chart.

[0055] First, in step S11, one or more input data points are provided, which are respectively composed of one or more sensor parameters and / or one or more default parameters as described above. To this end, a first set of input data points and a second set of input data points can be generated, and at the beginning of the method, the first set and the second set can be the same.

[0056] To this end, in step S12, as described above, for each input data point, a label is determined from one or more system parameters, so that a training data set or a validation data set is formed from the input data point and the assigned label, respectively. A set of multiple training data sets and a set of multiple validation data sets are obtained.

[0057] Now, with the aid of these training data sets or the first and second sets of supplementary training data sets as described below, the data-based model or the first or second data-based model is initially trained in step S13. The training is performed according to methods known per se, for example based on a back-propagation method.

[0058] Correspondingly, a first set of training data sets and a first set of validation data sets may be formed from the first input data points, and a second set of training data sets and a second set of validation data sets may be formed from the second input data points.

[0059] In step S14, it is checked whether further training data sets are needed to improve the data-based model according to the specified interruption criteria. The first or second set of validation data sets can be used, for example, to check the corresponding interruption criteria for model training. If this is the case (option: yes), the method is continued with step S15. Otherwise (option: no), the method is continued with step S22.

[0060] Then, in step S15, a plurality of additional sets of input data points are provided as candidates for one or more training data sets to be determined subsequently.

[0061] In step S16, one or more further first or second input data points are selected from the set of input data points by means of a specified first selection function and a specified second selection function, respectively.

[0062] In a subsequent step S17 , one or more further first input data points and one or more further second input data points are provided with labels on a test bench or by on-site measurements or by simulations in order to form further first and second training data sets.

[0063] In step S18 , the training data set is supplemented by adding a further first training data set to the first set of training data sets, and the training data set is supplemented by adding a further second training data set to the second set of training data sets.

[0064] In step S19, the data-based model is retrained or retrained using the first or second set of training data sets supplemented with a further first or second training data set.

[0065] In step S20, the trained first and second data-based models are evaluated. According to the above method, the evaluation is performed with the aid of evaluation sets of the test data set, which are generated from an existing validation data set with the aid of a resampling method.

[0066] By determining the quality measure (see above), a model quality and an uncertainty measure may be determined for the first and second data-based model, respectively, which are assigned to the corresponding selection function.

[0067] Subsequently, in step S21, the training method can be continued based on the selection function for which a significantly better model quality has been produced. In addition, the method is continued using the model of the first or second model whose model quality is evaluated as better. The corresponding other data-based model can be discarded.

[0068] By returning to step S16, the method can be re-executed by discarding poor selection functions and the associated data-based models and partially adding new selection functions to the corresponding remaining good selection functions. This can be repeated. In this way, the selection function can be continuously improved so that a significantly better model quality can be produced with a smaller number of training data sets relative to the workload of obtaining labels.

[0069] This significantly better model quality can be ascertained by determining a significance measure as a comparison value for the model quality and the uncertainty measure, for example by means of a weighted sum as described above.

[0070] Resampling does not improve the expected result of the test statistic (here, model performance). That is, the model based on the data is still selected, which has the highest model quality according to the chosen metric on the evaluation set of the test dataset. Resampling simply adds a confidence interval to the mean of the model quality so determined. That is, there is an interval within which the determined model quality will lie with a certain probability when the entire experiment is repeated.

[0071] The above method enables to determine a suitable selection function according to the achieved model quality and uncertainty measure. A selection function with an importance measure that can achieve high model quality can be provided for selection for subsequent training methods.

Claims

1. A computer-implemented method for evaluating a selection function for an active learning method for training a data-based model (21), the method comprising the following steps: - providing (S1, S2; S11, S12) a training data set and a validation data set, wherein the training data set and the validation data set respectively assign input data points to labels; - performing (S3) training of the data-based model (21) based on the training data set by means of an active learning method based on the selection function; - generating multiple evaluation sets of the test dataset by resampling from the validation dataset; - determining (S7) a model quality and an uncertainty measure of said model quality based on a statistical evaluation of the model performance of said data-based model based on a plurality of evaluation sets of the generated test data sets; - Based on said model quality and said uncertainty measure, keeping (S9) or discarding said selection function.

2. The method according to claim 1, wherein: The selection function used is confirmed or rejected depending on the model quality and the uncertainty measure, in particular depending on an importance measure that depends on the model quality and the uncertainty measure, in particular based on a threshold comparison.

3. The method according to claim 2, wherein: The importance measure is determined by means of a weighted sum of the model quality and the uncertainty measure.

4. The method according to any one of claims 1 to 3, wherein: The selection function used is used to create one or more additional input data points or to select one or more additional input data points from the provided input data points, wherein labels are determined for the additional input data points in order to determine a further training data set, wherein the further training data set is added to the existing training data set and further training of the data-based model is performed.

5. The method according to any one of claims 1 to 4, wherein: By means of the active learning method based on the selection function, training of the data-based model is performed multiple times based on the training data set.

6. The method according to any one of claims 1 to 5, wherein: The method is performed to train a first data-based model and a second data-based model by means of a first selection function and a second selection function different from the first selection function, wherein a resulting model quality and a corresponding uncertainty measure are determined for the first selection function and the second selection function, respectively, wherein the first or second selection function is selected based on a comparison based on the resulting model quality and the corresponding uncertainty measure in order to generate other training data sets and further train the data-based models accordingly.

7. The method according to any one of claims 1 to 6, wherein: The validation dataset is used to check the discontinuation criteria for training the data-based model (21) and / or to determine overfitting or underfitting.

8. The method according to any one of claims 1 to 7, wherein: The trained data-based model (21) is used to control, regulate or operate the technical system (1).

9. An apparatus for performing the method according to any one of claims 1 to 8.

10. A computer program product comprising instructions which, when the program is executed by at least one data processing device, cause the data processing device to carry out the steps of the method according to any one of claims 1 to 8.

11. A machine-readable storage medium comprising instructions which, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 8.

Images