Active data generation considering uncertainty

Abstract

The present invention relates to a method for generating data based on existing data with a single annotated data point, the method comprising: - training a first predictor based on the existing data; - determining a prediction error of the first predictor for each data point; - training a second predictor to determine an expected prediction error and an uncertainty of the first predictor; - determining a data description that maximizes the combination of the expected prediction error and the uncertainty; and - generating data based on the previously determined data description.

Description

Technical Field

[0001] The invention relates to a method for generating data based on existing data having a single annotated data point, a method for training a predictor with data generated in this way, a computer program and a data carrier. Background Art

[0002] In the field of machine learning, a predictor is an approximation of a multidimensional function.

[0003] Determining which data points in the overall data space to use for collection, generation, and use in an AI (artificial intelligence) method is often dependent on choices made by humans, prescribed criteria, or performed randomly.

[0004] Active data generation methods are necessary, for example, to enable a "co-learning" with domain experts in order to support (human) experts in iterative calibration or parameterization tasks. In such scenarios, the expert's task is to manually select and evaluate data points (e.g. system parameters) to achieve a given optimization goal. Active data generation methods are also necessary in a fully automatic data collection.

[0005] “Active learning” methods determine which data points should be annotated with a target value (such as category, position or KPI (key performance indicator)) and are based on an existing (unannotated) dataset. An alternative approach that also selectively determines which data points should be generated is described in “Counterexample-Guided Data Augmentation” by Tommaso Dreossi, Shromona Ghosh, Xiangyu Yao, Kurt Keutzer, Alberto Sangiovanni-Vincentelli, and Sanjit A. Seshia, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, July 2018, pp. 2071–2078. In addition, the article "A Theory of Formal Synthesis Learned by Induction" by S. Jha and S. A. Seshia, Acta Informatica (2017), Vol. 54, p. 693 (https: / / doi.org / 10.1007 / s00236-017-0294-5), describes a formal framework idea that covers a whole class of ideas without mentioning possible implementations.

[0006] The generation and collection of such data is time consuming, costly and partial, therefore improving the annotation process is not sufficient.

[0007] In the prior art, there is known a method of generating data which maximizes a prediction error of a predictor to generate new data based on a data description of data having a maximum prediction error.

[0008] According to the prior art, the selection of the data points to be generated is a task for an expert. Firstly, there are automated methods for selecting the data points to be generated, but these methods only consider which data points are (potentially) difficult to predict by an AI (artificial intelligence) method, without considering the reliability of the prediction. Therefore, it is not guaranteed that the expected or required data space is fully covered. In addition, these known methods can only be used for 0 / 1 errors. This 0 / 1 error does not indicate the degree of error, but only distinguishes between an error (=1) and an error-free (=0).

[0009] In particular, minimizing the data generation is of great importance in order to minimize the manual work of the experts. In addition, carefully selected training data can increase the reliability of the predictions and thus the quality of the AI ​​(artificial intelligence) method. By using a continuous error metric, an improvement in the quality or error assessment of the AI ​​(artificial intelligence) method can also be achieved. In this way, an active, guided data generation can be used, which takes into account both the optimization goal and the coverage of the data space.

[0010] Therefore, a method according to the present invention has the following steps: training a first predictor based on existing data, determining a prediction error of the first predictor for each data point, training a second predictor to determine an expected prediction error and uncertainty of the first predictor, determining a data description that maximizes an expected prediction error and uncertainty combination, and generating data based on the previously determined data description.

[0011] The amount of data required can be reduced by generating targeted data points that are specifically relevant to the problem and ensuring adequate coverage of the data space.

[0012] Using a Second Predictor The method can also be used in cases where the first predictor cannot be inverted or the predictor input space is not available for data generation.

[0013] Another method according to the invention trains a predictor using the data generated by the above method and also has the steps of checking whether the quality previously determined by the first predictor has been reached and annotating the new data and adding it to the existing data.

[0014] A computer program according to the invention comprises instructions for causing a computer system to execute one of the above methods. It has the advantage that the method can be applied particularly quickly and efficiently.

[0015] The above-mentioned computer program can be stored on a data carrier according to the present invention, which has the advantage that the computer program can be easily transmitted and copied.

[0016]

[0011] Other features of the present invention will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The figure shows a flow chart of a method for generating data according to the prior art;

[0018] Figure 2 The figure shows a flow chart of a method for generating data according to the present invention;

[0019] Figure 3 The figure shows a flow chart of a method for training a predictor, said predictor comprising a method for generating data according to the present invention;

[0020] Figure 4 The figure shows a schematic diagram of a testing device;

[0021] Figure 5 The figure shows a schematic diagram of a vehicle, and

[0022] Figure 6 The figure shows a schematic diagram of a data carrier according to the invention. DETAILED DESCRIPTION

[0023] Figure 1 A method according to the prior art for generating data suitable, for example, for training a predictor is shown. In this case, a predictor is trained based on annotated data in step A1 and a prediction error of the data is determined in step A2. Then, a description of a data point is determined in step A3 that maximizes the prediction error. Subsequently, one or more new data points are generated in step A4 based on the description.

[0024] exist Figure 2 1 shows a flow chart of a method according to the invention. The method according to the invention starts with step S1, whereby a first predictor is trained based on existing annotated data. The annotated data here consist in particular of data points relevant for the training and advantageously also include a description. Then, in step S2, a prediction error of the first predictor is determined for each of these data points based on at least part of the existing annotated data.

[0025] Then, in step S3, a second predictor is trained to determine an expected prediction error and an uncertainty of the first predictor. This is done in particular on the basis of the data point prediction errors of the first predictor. For this purpose, a description of the data points or a description of a data space / data region is also advantageously used.

[0026] Then, in step S4, a data description is determined by an algorithm that maximizes the combination of expected prediction error and uncertainty. The upper confidence limit (UCB) can be used as a measure in this regard, for example, which maximizes the sum of the expected prediction error and an uncertainty that can be expanded or compressed by a constant. Thus, the data description that maximizes the measure can be selected, for example, by randomly generating data descriptions.

[0027] Then, in step S5, new data are generated based on the data description previously determined in S4. The new data may, for example, describe at least a portion of a data space region in which the first predictor still has significant errors or is not covered by the first predictor.

[0028] A method for training a predictor by means of the method for generating data according to the invention comprises steps S1 to S5 and a step S2.1 and a step S6, and Figure 3 It is presented in the form of a flow chart.

[0029] In step S2.1, a check is performed to determine whether the quality of the first predictor is sufficient or optimal for the desired application. If the quality is sufficient or optimal, the method can be terminated here, otherwise the method is continued in step S3. The quality of the predictor is determined here with the aid of a suitable metric or another suitable calculation rule. For example, the quality of the first predictor can be generated from the arithmetic mean of the prediction errors of the single data points. The decision as to how high the quality of the first predictor should or must be is application-specific. Thus, for example, in safety-related applications, an error probability of less than 1% is generally required; for other applications, the consequences (e.g. increased costs) can be estimated by an estimator and a limit value can be determined that is sufficient to achieve the specified quality.

[0030] In step S6, the data generated or collected in S5 are annotated and added to the already existing annotated data. The method then continues in step S1.

[0031] The method according to the invention can be used, for example, to generate predictor data for predicting an engine performance based on engine parameters in order to train such a predictor. To this end, annotated data comprising engine parameters and measured performance data are used as a basis ( Figure 2 Here, the prediction error is determined ( Figure 2 Step S2) in includes the difference between the predicted performance and the actual measurement result.

[0032] The training of the predictor is now checked to see if it has achieved the desired prediction error. If so, the training method ends here ( Figure 3 Otherwise, a second predictor that can map uncertainty is trained ( Figure 2The second predictor is used to predict the expected performance difference based on the engine parameters. Here, the data space may include different possible areas, in particular the complete frequency spectrum, engine parameters and possible power values.

[0033] Then determine the data description including the engine parameters that maximize the second predictor ( Figure 2 Step S4 in ).

[0034] Generate new data based on specific data description ( Figure 2 Step S5) may here for example comprise a new evaluation of the given engine parameters, in particular further performance measurements, for example on an engine test bench. The new data may then, if necessary / optionally, be annotated and added to the existing data for training the predictor ( Figure 3 The method can be used to generate a predictor for an engine-driven vehicle that is particularly effective or accurate in determining engine performance / power based on other engine parameters. This can be used in a performance / power measurement station or in a vehicle and facilitates and optimizes performance / power determination.

[0035] The method according to the invention can also be used to generate data for a predictor that can find error locations on a printed circuit board image and to train such a predictor. Here, annotated images of printed circuit boards with known defects or errors are used as data. The prediction error of the first predictor can then be, for example, the deviation between the prediction error and the position of the known error on the printed circuit board. A second predictor can then be trained to predict the expected position deviation based on the printed circuit board description. The maximized printed circuit board description is then selected and new data is generated. For example, this can include creating one or more intentionally defective printed circuit boards with different errors and error locations that correspond to the printed circuit board description. For example, the predictor can be used in a test device for checking printed circuit board errors after a production process or during a production process. For this purpose, the test device can have a data carrier storing at least the predictor and also, in an advantageous manner, storing one of the computer programs according to the invention for implementing the method according to the invention for generating data or for training a predictor, in particular also storing data for training the predictor, the test device also comprising a computing unit equipped with the predictor, which is used to implement the training of the predictor and the generation of data, and the test device is also equipped with a camera intended for data recording. The testing device may include a communication unit for wired or wireless communication and may also be part of a network structure, in particular a client service architecture, a master-slave architecture, a cloud architecture, an Internet of Things architecture, so that the predictor, computer program and / or data do not have to exist continuously on the testing device itself.

[0036] The computing unit can control the test device, in particular the camera. The computing unit is also suitable for carrying out the method for generating data according to the invention and for training the predictor, in particular based on new data that can be recorded by the camera.

[0037] The test device, the predictor, the method for generating data and the method for training a predictor can also be used for other products and production processes, in particular for those that require low error rates and / or adaptability and flexibility, and in particular for products and production processes that can also quickly adapt to new and changing parameters or errors. This is particularly true in the fields of optical production, medical technology, drug production and chip manufacturing. Therefore, such a test device can also be used in the above-mentioned performance measurement bench or in a vehicle, in particular for performance / power measurement of engines and for face recognition in a vehicle.

[0038] Figure 4 Such a test device as schematically shown in FIG. 1 can also be a test device according to the invention. Here, the test device 1 comprises a sensor device, which is designed as a camera 2, a computing unit 3 and a data carrier 4. In addition, it can also include a communication device 5.

[0039] The method for generating data can thus generate data in a targeted manner in order to describe the data space as optimally as possible, so that systems with low hardware resources are also suitable for carrying out the method.

[0040] The method according to the invention can also be used to generate data for face recognition and to train a predictor for face recognition, in particular face recognition in vehicles. In this case, data points with associated annotations such as face position and descriptions such as light level, vehicle type, person type and accessories are available. A first predictor is trained using a subset of the data points, and the remaining data points are used to determine the difference between prediction and annotation. Then, to determine the face position, the deviation between the predicted and annotated face position is determined at the remaining data points.

[0041] By describing the data points and predicting the deviations from the annotations, the second predictor is trained to predict the deviations from the annotations with a certain uncertainty based on the description. The second predictor may include, for example, a Gaussian process that determines such predictions and uncertainties, such as in the form of confidence intervals.

[0042] A (random) description is then generated from the possible values ​​of the description. For this purpose, the expected prediction error and uncertainty are determined by the second predictor and the maximum or high value provided is selected from the prediction error and uncertainty combination. Based on the selected description, one or more data points corresponding to these descriptions are generated. For example, with the aid of rendering software, an image of a face that meets these descriptions can be created. In the images generated in this way, annotations can be implemented particularly simply in an advantageous manner. The annotated data points can then be added to the specified data points and the first predictor can be retrained.

[0043] The result is a predictor for determining the location of faces, wherein the quality of the prediction expected in new images based on the described method is maximized. Furthermore, the number of required (image) data points is minimized by the described method. Likewise, the predictor or other predictors can be trained to perform facial expression recognition and other forms of face recognition in an improved manner.

[0044] The method for generating data, the method for training a predictor and the predictor itself can be used in particular for face recognition in a vehicle. The vehicle can hereby comprise at least one camera or another sensor for generating new data. The vehicle can also comprise a computing unit for carrying out the method according to the invention for generating data, training the predictor and predictors, in particular based on new data which can also be recorded by the camera or the sensor. The vehicle can also comprise a data carrier on which data for training a predictor and a computer program for carrying out the method for generating data, training the predictor and predictors are stored.

[0045] Figure 5The vehicle shown is a vehicle of this type, in particular also a vehicle according to the invention. In this case, the vehicle 1 comprises at least one sensor, which is designed in this case in the form of a camera 2, a computing unit 3 and a data carrier 4. In addition, the vehicle can also comprise a communication unit 5. In this case, the camera 2, the computing unit 3, the data carrier 4 and / or the communication unit 5 can also be arranged together in one or more devices. Thus, for example, the computing unit 3, the data carrier 4 and / or the communication device 5 can be arranged in the housing of the camera 2.

[0046] In a vehicle, the method according to the invention described here can also be used for other devices that can be operated and improved with trained algorithms and data evaluation. This can be useful in particular when using a vehicle, especially an autonomous vehicle, in the case of rapidly changing situations and data volumes.

[0047] Figure 6 : is a schematic diagram of a data carrier 4 according to the invention. On the data carrier 4 at least a computer program according to the invention can be stored for implementing the method according to the invention for generating data and training a predictor, in particular data for training the predictor and the predictor itself can be stored.

[0048] Reference numerals list

[0049] A1–A5 Method Steps

[0050] S1–S6, S2.1 Method steps

[0051] 1 Inspection device

[0052] 2 Camera

[0053] 3 Computational Unit

[0054] 4 Data carrier

[0055] 5 Communication unit

[0056] 10 Vehicles

Claims

1. Methods for generating data based on existing data with a single annotated data point, where: The method includes: - training (S1) a first predictor based on existing data; - for each data point, determining (S2) a prediction error of the first predictor; - training (S3) a second predictor to determine an expected prediction error and an uncertainty of the first predictor; - determining (S4) a data description that maximizes the expected prediction error and uncertainty combination; and - Generating (S5) data based on the previously determined data description.

2. A method for training a predictor using data generated by the method according to claim 1, further comprising: - checking (S2.1) whether a previously determined first predictor quality has been reached; as well as - Annotate the new data and add (S6) to the existing data.

3. Computer program comprising instructions for causing a computer system to execute the method according to any one of claims 1 or 2.

4. A data carrier having stored thereon a computer program according to claim 3 for executing the method according to any one of claims 1 or 2.

5. A test device (1) comprising a sensor device (2), a computing unit (3) and a data carrier (4), wherein the test device executes the method for generating data according to claim 1 or the method for training a predictor according to claim 2.

6. Testing device according to claim 5, comprising a communication device (5).

Images