Method for validating machine learning algorithms

Abstract

The invention relates to a method for verifying a machine learning algorithm, the machine learning algorithm being trained to identify an object in image data, and wherein the method (1) comprises: providing a machine learning algorithm, the machine learning algorithm being trained to identify an object (2) in image data; generating tagged verification data for verifying the machine learning algorithm, the verification data each containing at least one disturbance variable (3); and verifying the machine learning algorithm (4) on the basis of the generated verification data.

Description

Technical Field

[0001] The present invention relates to a method for validating a machine learning algorithm and in particular to a method by which a trained machine learning algorithm can be verified in a simple manner and with low resource consumption how it reacts to disturbance variables. Background Art

[0002] Machine learning algorithms are based on the use of statistical methods to train a data processing system so that it can perform a specific task without the data processing system having been originally explicitly programmed for this task. The goal of machine learning is to build algorithms that can learn from data and make predictions. These algorithms create mathematical models with which data can be classified, for example.

[0003] Robustness is also understood to be the property of a method or system that it functions reliably even under unfavorable conditions. In machine learning algorithms, robustness describes the sensitivity to interference variables such as noise. With low robustness, even small interference variables can lead to high error rates in the processing of the input data by the corresponding machine learning algorithm, while high robustness provides correspondingly stable results even with greater interference.

[0004] As part of quality assurance during the manufacturing process, the quality state of the test object or the produced technical component is usually also tested. In this case, one or more quality features that represent the quality state of the technical component are evaluated. Such quality features can include, for example, shape or surface properties, wherein it is to be determined whether anomalies exist in this regard. Anomalies usually refer to deviations from specified specifications or target properties. On the smooth surface of the manufactured component, scratches, for example, may represent such anomalies. Based on the determination of the quality state, it can then be decided, for example, whether the corresponding technical component can be processed further or is to be discarded, for example automatically.

[0005] In order to be able to implement such tests reliably and independently of human senses, it is known to use machine learning algorithms, especially machine learning algorithms that are trained to determine or derive the quality status of the technical component from the image data of the display technical component. However, during the corresponding manufacturing process, influences may occur, which are reflected in the corresponding image data of the test object as interference quantities. This influence may be, for example, vibration, air humidity, or it may also be dust. Vibration can be reflected in the image data as image noise, for example. Air humidity can, for example, cause local light refraction by forming small water droplets on the camera lens, which produces blind spots. On the other hand, dust can, for example, reduce image quality in the form of gray fog or in the form of identifiable interference bodies. Here, if the corresponding machine learning algorithm is not robust enough for such interference quantities, such interference quantities may cause: the technical component is mistakenly discarded, which leads to unnecessary costs, or else cause: the component is mistakenly not discarded, which may lead to safety risks when further processing and / or using the technical component.

[0006] An artificial intelligence module is known from the publication DE 102017218889 A1, which is designed to: process one or more input parameters into one or more output parameters through an internal processing chain, wherein the internal processing chain is specified by one or more parameters, and wherein the artificial intelligence module is designed to: determine the one or more parameters from one or more storage values ​​stored in a memory, wherein a distribution module is provided, which is designed to extract single values ​​from one or more statistical distributions and determine the one or more parameters therefrom, wherein at least one statistical characteristic parameter of each statistical distribution depends on at least one storage value. Summary of the invention

[0007] The object of the present invention is therefore to specify a method by means of which it is possible to verify how a trained machine learning algorithm reacts to disturbance variables.

[0008] This object is achieved by a method for validating a machine learning algorithm according to the features of independent patent claim 1 .

[0009] Furthermore, the object is achieved by a system for validating a machine learning algorithm according to the features of independent patent claim 6.

[0010] According to one embodiment of the invention, the object is achieved by a method for validating a machine learning algorithm, wherein the machine learning algorithm is trained to recognize objects in image data, and wherein the method comprises:

[0011] providing a machine learning algorithm, the machine learning algorithm being trained to recognize objects in the image data;

[0012] generating labeled validation data for validating the machine learning algorithm, wherein the validation data each comprises at least one disturbance variable; and

[0013] Based on the generated validation data, the machine learning algorithm is validated.

[0014] Image data are understood here to be data which can be reproduced as images or graphics with the aid of special programs.

[0015] Furthermore, labeled verification data are understood to be image data that are labeled or provided with ground truth information, which are used to test or verify a machine learning algorithm and which are generally not part of the training data for training the corresponding machine learning algorithm.

[0016] In this case, comparison data containing interference variables are understood to mean corresponding images or verification data that are influenced by the interference variables.

[0017] Thus, a method is described which, in a simple manner and without complex and resource-intensive adjustments, provides reliable conclusions about the extent to which a relevant trained algorithm is prone to errors in carrying out the task for which it was trained in an environment in which at least one disturbance variable is present, in particular a disturbance variable which is not contained in the corresponding training data. This makes it possible, for example, to select a sufficiently robust trained algorithm in order to thereby ensure the desired process safety in determining the quality state of a technical component during a manufacturing process, whereby safety risks can be avoided.

[0018] Thus, in summary, a method was described by which it is possible to verify how a trained machine learning algorithm reacts to the amount of disturbance.

[0019] In this case, the step of generating the labeled verification data may include generating the labeled verification data using a generative adversarial network.

[0020] A generative adversarial network is understood to be a machine learning algorithm that is based on two interconnected artificial neural networks. In this case, a first artificial neural network, the generator, generates data that initially consists only of random statistical noise. A second neural network, the discriminator, analyzes or classifies the data generated by the first artificial neural network, wherein the discriminator is trained with real or actual data, i.e., ground truth information, in order to be able to evaluate these data. In this case, the generative adversarial network is usually trained by an iterative process, wherein the generator and the discriminator are trained alternately or in each iteration step. What is to be achieved by this is that the trained generative adversarial network can then imitate the real situation as well as possible.

[0021] Thus, verification data can be generated in a simple manner by means of known algorithms without the need for complex and resource-intensive adaptations.

[0022] However, the step of generating the labeled verification data has the following features: generating the labeled verification data by using a generative adversarial network is only one possible implementation. Then, the verification data can also be generated by other known tools or functions for generating image data containing interference.

[0023] In one embodiment, the step of validating the machine learning algorithm further comprises:

[0024] For all generated verification data, respectively determine a robustness value based on reference truth information about the corresponding verification data, the size of the corresponding at least one interference quantity, and at least one output value of the machine learning algorithm for the corresponding verification data;

[0025] Determining a robustness value of the machine learning algorithm based on the robustness values ​​of all generated validation data; and

[0026] The robustness value of the machine learning algorithm is compared to a threshold value of the machine learning algorithm.

[0027] In this context, the magnitude of the respective interference variable is understood to mean a value which describes the degree of the respective interference variable or of the interference variable influencing the respective verification data, or the degree of the respective interference variable.

[0028] Furthermore, the output value of the machine learning algorithm for the corresponding verification data is understood to be: the output value of the machine learning algorithm generated based on the corresponding verification data.

[0029] In this case, a robustness value is understood to be a value which characterizes or describes the robustness of the respective result.

[0030] Furthermore, the threshold value of the machine learning algorithm is understood in particular to mean a threshold value which the robustness value of the respective machine learning algorithm should not fall below.

[0031] Thus, the machine learning algorithm can be validated in a simple manner and with low resource requirements, such as requirements for memory and / or processor power.

[0032] Furthermore, labeled verification data can be generated from the sensor data detected by the sensor, wherein the labeled verification data relate to different orientations of the sensor.

[0033] A sensor, also called a (measuring) detector, is a technical component that can qualitatively detect specific physical or chemical properties and / or material texture of the surrounding environment of the technical component, or can quantitatively detect specific physical or chemical properties and / or material texture of the surrounding environment of the technical component as measurement parameters.

[0034] In this case, alignment of a sensor is understood to mean adjusting the orientation and / or the position of the sensor.

[0035] Thus, in a simple manner, conditions outside the actual data processing system on which the machine learning algorithm is validated can be detected and taken into account when validating the machine learning algorithm.

[0036] Using another embodiment of the present invention, a method for determining the quality status of a technical component by means of a machine learning algorithm is also described, wherein the machine learning algorithm is trained to: determine the quality status of the technical component based on image data displaying the technical component, wherein the method comprises: providing image data displaying the technical component; providing a machine learning algorithm, wherein the machine learning algorithm is trained to: determine the quality status of the technical component based on the image data displaying the technical component, wherein the machine learning algorithm is verified by the above-mentioned method for verifying a machine learning algorithm; and determining the quality status of the technical component based on the provided image data and the provided machine learning algorithm.

[0037] In this case, the quality state is understood to be a variable which describes the quality or at least one quality feature of the technical component.

[0038] Thus, a method for determining the quality state of a technical component by means of a machine learning algorithm is described, which method is based on a machine learning algorithm that is verified by means of a method that can be used to verify how the trained machine learning algorithm reacts to disturbance variables. In particular, the machine learning algorithm is verified by means of a method that, in a simple manner and without complex and resource-intensive adjustments, provides reliable conclusions about the extent to which the relevant trained algorithm is prone to errors in performing the task for which it was trained in an environment in which at least one disturbance variable occurs, in particular a disturbance variable that is not contained in the corresponding training data. This makes it possible, for example, to select a sufficiently robust trained algorithm in order to thereby ensure the desired process safety in determining the quality state of the technical component during the manufacturing process, thereby avoiding safety risks.

[0039] Furthermore, a system for validating a machine learning algorithm is described with a further embodiment of the present invention, wherein the machine learning algorithm is trained to recognize objects in image data, wherein the system has:

[0040] a providing unit, the providing unit being configured to provide a machine learning algorithm, the machine learning algorithm being trained to recognize objects in the image data;

[0041] A generating unit, the generating unit being configured to generate labeled verification data for verifying the machine learning algorithm, wherein the verification data respectively contain at least one interference quantity; and

[0042] A verification unit is designed to verify the machine learning algorithm based on the generated verification data.

[0043] Thus, a system is described, with which it is possible to verify how a trained machine learning algorithm reacts to disturbance variables. In particular, a system is described which, in a simple manner and without complex and resource-intensive adjustments, provides reliable conclusions about the extent to which the relevant trained algorithm is prone to errors in carrying out the task for which it was trained in an environment in which at least one disturbance variable occurs, in particular a disturbance variable that was not contained in the corresponding training data. This makes it possible, for example, to select a sufficiently robust trained algorithm in order to thereby ensure the desired process safety in determining the quality state of a technical component during a manufacturing process, whereby safety risks can be avoided.

[0044] In this case, the generation unit can be designed to generate labeled verification data using a generative adversarial network. Thus, the verification data can be generated in a simple manner by known algorithms without complex and resource-intensive adjustments.

[0045] However, the generation unit is designed to generate labeled verification data using a generative adversarial network, which is only one possible implementation. Then, the verification data can also be generated by other known tools or functions for generating image data containing interference.

[0046] In one embodiment, the verification unit has: a first determination unit, which is designed to: for all generated verification data, based on the reference truth information about the corresponding verification data, the size of the corresponding at least one interference quantity, and the output value of the machine learning algorithm for the corresponding verification data, respectively determine the robustness value; a second determination unit, which is designed to: determine the robustness value of the machine learning algorithm according to the robustness values ​​of all generated verification data; and a comparison unit, which is designed to: compare the robustness value of the machine learning algorithm with a threshold value of the machine learning algorithm. Therefore, the machine learning algorithm can be verified in a simple manner and with low resource requirements, such as memory and / or processor power requirements.

[0047] Furthermore, the generation unit can be designed to generate labeled verification data from the sensor data detected by the sensor, wherein the labeled verification data relate to different orientations of the sensor. Thus, conditions outside the actual data processing system on which the machine learning algorithm is verified can be detected in a simple manner and taken into account when verifying the machine learning algorithm.

[0048] In addition, using another embodiment of the present invention, a system for determining the quality status of a technical component by means of a machine learning algorithm is described, wherein the machine learning algorithm is trained to determine the quality status of the technical component based on image data displaying the technical component, wherein the system comprises: a first providing unit, which is designed to provide image data displaying the technical component; a second providing unit, which is designed to provide a machine learning algorithm, which is trained to determine the quality status of the technical component based on image data displaying the technical component, wherein the machine learning algorithm is verified by the above-mentioned system for verifying the machine learning algorithm; and a determination unit, which is designed to determine the quality status of the technical component based on the provided image data and the provided machine learning algorithm.

[0049] Thus, a system for determining the quality state of a technical component by means of a machine learning algorithm is described, the system being based on a machine learning algorithm verified by means of a system that can be used to verify how the trained machine learning algorithm reacts to disturbance variables. In particular, the machine learning algorithm is verified by means of a system that, in a simple manner and without complex and resource-intensive adjustments, provides reliable conclusions about the extent to which the relevant trained algorithm is prone to errors in carrying out the task for which it was trained in an environment in which at least one disturbance variable is present, in particular a disturbance variable that was not contained in the corresponding training data. This makes it possible, for example, to select a sufficiently robust trained algorithm in order to thereby ensure the desired process safety in determining the quality state of the technical component during the manufacturing process, thereby avoiding safety risks.

[0050] In summary, it should be emphasized that the present invention describes a method for validating a machine learning algorithm and in particular a method that can be used to verify how a trained machine learning algorithm reacts to disturbance variables in a simple manner and with low resource consumption.

[0051] The described embodiments and refinements can be combined with one another as desired.

[0052] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described above or below with reference to the exemplary embodiments that are not explicitly mentioned. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are intended to provide a further understanding of the embodiments of the present invention. These drawings illustrate the embodiments and, together with the description, are used to explain the principles and design of the present invention.

[0054] Further embodiments and a number of the advantages mentioned can be derived with reference to these drawings. The elements represented in these drawings are not necessarily shown to scale with respect to one another.

[0055] in:

[0056] Figure 1 A flowchart showing a method for validating a machine learning algorithm according to an embodiment of the present invention; and

[0057] Figure 2 A schematic block diagram of a system for validating a machine learning algorithm according to an embodiment of the present invention is shown.

[0058] In the figures of the drawings, the same reference numerals denote identical or functionally identical elements, components or assemblies unless otherwise indicated. DETAILED DESCRIPTION

[0059] Figure 1 A flow chart of a method 1 for validating a machine learning algorithm according to an embodiment of the present invention is shown.

[0060] It is well known that machine learning algorithms based on deep learning, especially machine learning algorithms for image processing based on deep learning, are very sensitive to external interference or manipulation. Therefore, there is a need for methods that can be used to verify how trained machine learning algorithms react to interference amounts, especially such interference or manipulation.

[0061] Here, Figure 1 A method 1 for validating a machine learning algorithm is shown, wherein the machine learning algorithm is trained to recognize objects in image data, and wherein the method has:

[0062] Step 2: Provide a machine learning algorithm that is trained to recognize objects in the image data;

[0063] Step 3: generating labeled validation data for validating the machine learning algorithm, wherein the validation data each contain at least one interference quantity; and

[0064] Step 4: Validate the machine learning algorithm based on the generated validation data.

[0065] therefore, Figure 1 A method is shown which provides reliable conclusions in a simple manner and without complex and resource-intensive adjustments about the extent to which a relevant trained algorithm is prone to errors in carrying out the task for which it was trained in an environment in which at least one disturbance variable is present, in particular a disturbance variable which is not contained in the corresponding training data. This makes it possible, for example, to select a sufficiently robust trained algorithm in order to thereby ensure the desired process reliability in determining the quality state of a technical component during a manufacturing process, whereby safety risks can be avoided.

[0066] So, in summary, Figure 1 A method 1 is shown by which it is possible to verify how a trained machine learning algorithm reacts to disturbance amounts.

[0067] In this case, method 1 can be used to check in particular whether the algorithm is robust to interfering factors and whether it always indicates correct output values ​​in the presence of certain results. In particular, a quality gate is provided, for example a quality gate for a machine learning algorithm for deep learning training.

[0068] In this case, the at least one interfering variable can in particular be an interfering variable which occurs during the manufacturing process of the technical component. Such interfering variables can be, for example, vibrations, air humidity or also dust. Vibrations can be reflected in the corresponding image data, for example, as image noise. Air humidity can, for example, lead to local light refractions by the formation of small water droplets on the camera, which produce blind spots in the image data. On the other hand, dust can, for example, reduce the image quality in the form of gray fog or in the form of recognizable interfering bodies.

[0069] Here, the at least one interference variable may also be at least one interference variable that characterizes the purpose or use case in which the machine learning algorithm is to be used.

[0070] The machine learning algorithm may also be an artificial neural network, for example.

[0071] according to Figure 1 In an embodiment of the present invention, step 3 of generating labeled verification data further comprises: generating labeled verification data using a generative adversarial network.

[0072] like Figure 1 As also shown, step 4 of verifying the machine learning algorithm also has: step 5: for all generated verification data, based on the benchmark truth information about the corresponding verification data, the size of at least one corresponding interference quantity, and the output value of the machine learning algorithm for the corresponding verification data, respectively determine the robustness value; step 6: determine the robustness value of the machine learning algorithm based on the robustness values ​​of all generated verification data; and step 7: compare the robustness value of the machine learning algorithm with the threshold of the machine learning algorithm.

[0073] Here, the threshold of the machine learning algorithm may be based on the requirements of the purpose or use case in which the machine learning algorithm is to be used. For example, the more safety-critical the subsequent use case is, the higher the threshold may be selected.

[0074] The magnitude of the at least one interference variable may also be measured or detected during the generation of the corresponding verification data, for example by monitoring a corresponding parameter during the generation of the verification data.

[0075] The output value of the machine learning algorithm for the corresponding validation data may also be the corresponding prediction probability.

[0076] Furthermore, the robustness value of the machine learning algorithm may be formed, for example, by taking the average of the robustness values ​​of all generated validation data.

[0077] In addition, according to Figure 1 In an embodiment of the present invention, the marked verification data are generated from sensor data detected by the sensor, wherein the marked verification data relate to different orientations of the sensor.

[0078] In this case, a corresponding validated machine learning algorithm can be trained based on corresponding labeled training data to determine the quality state of a technical component, wherein the technical component can be produced, for example, during a manufacturing process, and wherein technical components that do not meet a specified quality standard can be automatically discarded. In addition, however, a corresponding machine learning algorithm can also be trained based on labeled training data to control a controllable system or a robotic system.

[0079] Figure 2 A schematic block diagram of a system 10 for validating a machine learning algorithm according to an embodiment of the present invention is shown.

[0080] Figure 2 In particular, a system 10 for validating a machine learning algorithm is shown, wherein the machine learning algorithm is trained to recognize objects in image data.

[0081] Here, if Figure 2 As shown, the system 10 has:

[0082] A providing unit 11, the providing unit is designed to provide a machine learning algorithm, the machine learning algorithm is trained to recognize objects in image data;

[0083] A generating unit 12, the generating unit being configured to generate labeled verification data for verifying the machine learning algorithm, wherein the verification data respectively contain at least one interference quantity; and

[0084] A verification unit 13, which is designed to verify the machine learning algorithm based on the generated verification data.

[0085] Here, the providing unit can be in particular a receiver which is designed to receive the corresponding data.In addition, the generating unit and the verifying unit can be implemented, for example, based on codes which are stored in a memory and can be executed by a processor.

[0086] according to Figure 2 In an embodiment, the generating unit 12 is further designed to generate labeled verification data by using a generative adversarial network.

[0087] like Figure 2 As also shown, the verification unit 13, according to Figure 2 The embodiment further comprises:

[0088] A first determination unit 14, the first determination unit being configured to: for all generated verification data, determine a robustness value based on reference truth information about the corresponding verification data, a size of the corresponding at least one interference quantity, and an output value of the machine learning algorithm for the corresponding verification data;

[0089] A second determination unit 15, the second determination unit is designed to: determine the robustness value of the machine learning algorithm according to the robustness values ​​of all generated verification data; and

[0090] A comparison unit 16 is configured to compare the robustness value of the machine learning algorithm with the interference factor of the machine learning algorithm.

[0091] Here, the first determination unit, the second determination unit, and the comparison unit may be implemented based on codes stored in a memory and executable by a processor, for example.

[0092] According to the embodiment of FIG. 3 , the generating unit 12 is further designed to generate the marked verification data from the sensor data detected by the sensor, wherein the marked verification data relate to different orientations of the sensor.

[0093] In this case, the sensor can in particular be an optical sensor, for example a camera.

[0094] Here, the system 10 can also be designed to: execute the above-mentioned method for verifying the machine learning algorithm.

Claims

1. A method for determining the quality status of a technical component by means of a machine learning algorithm, wherein the machine learning algorithm is trained to determine the quality status of the technical component based on image data showing the technical component, wherein: The method comprises the following steps: - providing image data showing said technical component; - providing a machine learning algorithm, the machine learning algorithm being trained to determine the quality status of the technical component based on image data showing the technical component, wherein the machine learning algorithm is validated by a method for validating a machine learning algorithm; and - determining the quality status of the technical component based on the provided image data and the provided machine learning algorithm, 2. Among them, The method for validating a machine learning algorithm comprises: providing a machine learning algorithm that is trained to recognize objects in image data (2); generating labeled validation data for validating the machine learning algorithm, wherein the validation data each contain at least one disturbance variable (3); and validating the machine learning algorithm (4) based on the generated validation data, and wherein, based on the validation result, a sufficiently robust trained machine learning algorithm is selected in order to ensure the desired process reliability with respect to determining the quality state of a technical component during a manufacturing process. The method according to claim 1, wherein the step of generating the labeled validation data (3) comprises generating the labeled validation data using a generative adversarial network.

3. The method according to claim 1 or 2, wherein: The step (4) of verifying the machine learning algorithm further comprises the following steps: - for all generated verification data, respectively determine a robustness value (5) based on reference truth information about the corresponding verification data, the size of the corresponding at least one interference quantity, and the output value of the machine learning algorithm for the corresponding verification data; - determining a robustness value of the machine learning algorithm based on the robustness values ​​of all generated validation data (6); and - Comparing the robustness value of the machine learning algorithm with a threshold value of the machine learning algorithm (7).

4. The method according to any one of claims 1 to 3, wherein: The labeled verification data are generated from sensor data detected by a sensor, and wherein the labeled verification data relate to different orientations of the sensor.

5. A system for determining the quality status of a technical component by means of a machine learning algorithm, the machine learning algorithm being trained to determine the quality status of the technical component based on image data showing the technical component, and wherein: The system has: a first providing unit, the first providing unit being designed to: provide image data showing the technical component; a second providing unit, the second providing unit being configured to provide a machine learning algorithm, the machine learning algorithm being trained to: determine a quality state of the technical component based on image data showing the technical component, wherein the machine learning algorithm is verified by a system for verifying machine learning algorithms; and a determination unit, which is designed to determine a quality state of the technical component based on the provided image data and the provided machine learning algorithm, The system (10) for verifying a machine learning algorithm comprises: A providing unit (11), the providing unit being configured to provide a machine learning algorithm, the machine learning algorithm being trained to recognize objects in image data; a generating unit (12), the generating unit being configured to generate labeled verification data for verifying the machine learning algorithm, wherein the verification data each comprises at least one interference quantity; and a verification unit (13) designed to verify the machine learning algorithm based on the generated verification data, and Therein, based on the validation results, a sufficiently robust trained machine learning algorithm is provided in order to ensure the desired process reliability with regard to determining the quality state of technical components during the manufacturing process.

6. The system according to claim 5, wherein: The generating unit (12) is designed to generate the labeled verification data by using a generative adversarial network.

7. The system according to claim 5 or 6, in, The verification unit (13) has: A first determination unit (14), the first determination unit being configured to: for all generated verification data, determine a robustness value based on reference truth information about the corresponding verification data, the size of the corresponding at least one interference quantity, and an output value of the machine learning algorithm for the corresponding verification data; A second determination unit (15), the second determination unit being configured to: determine a robustness value of the machine learning algorithm according to the robustness values ​​of all generated verification data; and A comparison unit (16), wherein the comparison unit is designed to compare the robustness value of the machine learning algorithm with a threshold value of the machine learning algorithm.

8. A system according to any one of claims 5 to 7, wherein: The generating unit (12) is designed to generate the marked verification data from sensor data detected by a sensor, wherein the marked verification data relate to different orientations of the sensor.

Images