Method and workflow system for expanding training data sets for machine learning

Abstract

Workflow for training a classifier for quality inspection in measurement technology. A computer-implemented method for expanding a training data set for machine learning is proposed. In this case, the method comprises providing a data set of an object to be studied. In this case, the data set contains coordinate values ​​and a measurement value for each coordinate. The method also comprises: identifying anomalies in a partition of the data set corresponding to a sub-area of ​​the object to be studied, and classifying the anomalies into a plurality of predefined classification classes by means of a first machine learning model trained with a first training data set. The method also comprises: determining a difference value of the anomaly compared to the trained first machine learning model, and when the difference value is above a first predefined threshold, adding data related to the identified anomaly to the first training data set.

Description

Technical Field

[0001] The present invention relates to a workflow for annotating a training dataset, and in particular to a computer-implemented method for extending a training dataset for machine learning, and to a corresponding workflow system. Background Art

[0002] Industrial production processes require constant quality control in order to consistently meet the requirements of consumers (industrial companies and individuals). For consistently high quality, even the slightest deficiencies are not tolerated on the market. In order to meet the requirements for consistently high quality of the manufactured products, automated quality assurance systems are often used. Some of these systems are based on purely optical methods, while others rely on mechanical / electromechanical measurements of product parameters. Another class of quality measurement methods relies on the evaluation of data by electron microscopes, fluorescence microscopes, light microscopes, optical coherence tomography systems, interferometers, spectrometers or other computer tomography systems.

[0003] Artificial intelligence techniques, in particular machine learning systems, can be used to assess the measured values. In order for these systems to function as desired, it is usually necessary to have a large amount of training data in order to detect anomalies in the produced workpieces using the machine learning system that is then trained. All training data must be annotated manually, which requires a significant investment of time by professionals. Annotation in this context is understood to mean the process of assigning each anomaly to be classified to a desired, predetermined anomaly class. This could be a porosity, a core break, a wall displacement, etc.

[0004] This manual process is problematic from many perspectives: collecting the data is very time-consuming, as is the manual annotation process. One problem with collecting the data is that the frequency distributions of defect classes often vary widely. In order to ensure that there are enough examples of even relatively rare defects for training purposes, extremely large amounts of data often need to be recorded. However, this is technically very complex and expensive. In particular, it is often not possible to store all anomalies.

[0005] Another problem with data annotation is that in some cases very similar or even redundant data are recorded and stored. If these redundant data are annotated without proper selection, one can expect hardly any improvement in the classification performance, since the redundantly annotated data are more or less irrelevant. Especially in the context of (feasibility) studies, it is often possible to record a large amount of data; however, in case of poor data sampling, it may be the case that a large number of similar data points are unnecessarily collected and annotated in a time-consuming manner.

[0006] The set of defect features is usually incomplete. Instead, new defect classes often need to be recorded in the existing catalog over time. Possible reasons for this are changes in the parameters of the supervised process or the determination that the original training data set selected is too small and therefore not fully representative of the overall quality assurance task. In this respect, therefore, it is necessary to prevent previously unknown defects from being assigned to known defect classes (classification) in the classification or to prevent features of known classes from being incorrectly classified as non-defects or defects of another class. When selecting new examples, it should be avoided to select irrelevant outliers for annotation. Furthermore, the selection and annotation process should provide the possibility that the operator / visual inspector can influence the annotation process, for example, because the selected examples result from an erroneous recording and are not assigned to any relevant defect type.

[0007] Third, if the selected defects are not annotated by application professionals but by trained non-professionals, mismatches or errors in the annotations should be avoided.

[0008] Furthermore, in real applications, incremental adaptation of the classifier after adding annotations is usually not feasible, since the application expert would like to annotate, for example, a few items of data (e.g., 30-300) at a time on a workpiece, since training takes a relatively long time compared to the sample inspection time. Therefore, it is feasible to perform extended training of the classifier with only a few new data points. Therefore, when selecting a few examples for annotation, one must not only ensure that each individual example provides as much information as possible, but also that the combination of these examples (i.e., training data) is not redundant.

[0009] Additionally, it is difficult to annotate 3-dimensional data due to the challenges of viewing it on a 2D display. Additionally, classifying the data is very challenging due to the large volume of data. In the presence of a large number of sub-volumes to be classified, a correspondingly large amount of computing power is required.

[0010] Therefore, the basic goal of the concept proposed here is to optimize the annotation process elegantly, reduce the burden on the responsible personnel, enable continuous additions to the training dataset and allow appropriate opportunities to respond only to anomalies that cannot be correctly classified. Summary of the invention

[0011] This object is achieved by the computer-implemented method according to the invention for expanding a training data set for machine learning and a corresponding workflow system for expanding a training data set for machine learning proposed here.

[0012] According to a first aspect of the present invention, a computer-implemented method for expanding a training data set for machine learning is proposed. The method comprises providing a data set of an object to be studied, wherein the data set has coordinate values ​​and a measurement value of each coordinate.

[0013] The method also includes: identifying anomalies in a partition in the data set corresponding to the sub-region of the object to be studied, and classifying the anomalies into a plurality of predefined classification classes by a first machine learning model trained with a first training data set.

[0014] The method also includes determining a difference value of the anomaly compared to the trained first machine learning model, and when the difference value is above a first predefined threshold, adding data related to the identified anomaly to the first training data set.

[0015] According to a second aspect of the present invention, a corresponding workflow system is proposed, which has:

[0016] a receiving unit, which is designed to receive a data set of the object to be investigated, wherein the data set has coordinate values ​​and a measured value for each coordinate,

[0017] a detection module designed to identify anomalies in a partition of the data set corresponding to a sub-region of the object to be studied,

[0018] - a classification system designed to classify the anomaly into a plurality of predefined classification classes by means of a first machine learning model trained with a first training data set,

[0019] a difference determination module configured to determine a difference of the anomaly compared to a trained first machine learning model, and

[0020] - an adding unit designed to add data related to the identified anomaly to the first training data set when the difference is above a first predefined threshold.

[0021] Furthermore, the embodiments may be implemented in the form of a corresponding computer program product accessible by a computer-usable or computer-readable medium containing program code for use by or in conjunction with a computer or instruction execution system. In the context of this specification, a computer-usable or computer-readable medium may be any device having means for storing, communicating, forwarding or transmitting a program for use by or in conjunction with an instruction execution system, a corresponding device or apparatus.

[0022] The computer-implemented method of expanding a training data set for machine learning has the following advantages and technical effects:

[0023] First, the proposed concept allows for a significant reduction in annotation costs and effort. To reduce the amount of data and, therefore, the complexity, two-dimensional XY, YZ and XZ sections through the center of the anomaly can be used instead of the full 3D volume for annotation and analysis / classification. In this case, the relevant sections can be obtained from the probability map of anomaly detection. As shown experimentally, the cross-sectional images contain enough information to distinguish between error classes, but are easier to handle in terms of processing than the full 3D volume information. As an alternative, other 2D projections of the 3D volume (e.g., local minimum / maximum projections) can also be used.

[0024] Furthermore, training time can be reduced. Since, as a result of the targeted selection of data, less data overall leads to the same information content, less time is needed overall for classifier training. This has a direct impact on faster turnaround, which also allows interactive switching between training, data selection and annotation, allowing solutions to be implemented faster.

[0025] Furthermore, the concept proposed here leads to smaller classification errors. Since defects that are difficult to detect are preferably selected for annotation, the defect classification is able to reliably and quickly detect even difficult anomalies. The classification accuracy is thus increased in a sustainable manner, which improves the results of the quality inspection and thus the results of the relevant products.

[0026] Furthermore, manual intervention can be carried out in an automated or partially automated annotation process in order to thereby prevent incorrect annotation parameters and to adjust them accordingly.

[0027] Overall, the annotation process is thus elegantly optimized, reduces the burden on responsible personnel, allows continuous additions to the training dataset, and provides an appropriate opportunity to respond only to anomalies that cannot be correctly classified.

[0028] Further embodiments of the inventive concept of the method are presented below, which can be used in the same way for corresponding workflow systems:

[0029] According to one embodiment of the method, the anomaly may be a large number of anomalies, and the adding may include: when the difference is above a second predefined threshold, adding data from the selected identified anomaly to the training data set. In this case, the first threshold may draw the operator's attention to the fact that the anomaly has not been clearly assigned to a class and manual intervention in the annotation process seems to be necessary.

[0030] However, if the determined difference value is higher than the second difference value, for example a higher value, it may be assumed that a significantly different anomaly is present, which may then also be automatically applied for annotating the training data set.

[0031] It is also worth noting that the data on anomalies may not only consist of one scan data set, but may also contain a large amount of data combined from several scans, thereby greatly increasing the probability of detection.

[0032] According to another embodiment of the method, the difference can be determined by a novelty degree method for the selected anomaly. The novelty degree method can be selected from the group consisting of: novelty detection, estimation error reduction, estimation entropy minimization, expected model output change (EMOC), MC-dropout, OpenMAX (the latter two are used for neural networks), SVM margin, neural network, the ratio of the highest classification probability to the second highest classification probability, classification probability entropy, GP variance, the variance of each classification probability from an aggregate classifier (e.g., bagging), the variance of the classification probability in the case of interference of the input signal (i.e., in the case where the data set is related to the object to be studied).

[0033] This provides a wide library of methods for the difference, from which the person responsible for the workflow can choose without artificial restrictions. The method for detecting the difference can be further supplemented by those skilled in the art.

[0034] According to a further possible embodiment of the method, a combination of novelty degree methods can be determined using a method selected from the group consisting of: multi-armed bandit formulation, success-driven selection among multiple criteria, linear combination based on reward functions, reinforcement learning. These variants make it possible to consider further more complex novelty degree methods in order to thus satisfy the combination of identification anomalies and classification used.

[0035] According to a further embodiment, when determining the difference, the method may further comprise determining the difference or the combined difference by a second machine learning model trained with a second training data set. In particular, this may be one of the methods of forest regression, linear regression, Gaussian process, neural network, etc. Such an accurate qualitative check may more accurately detect whether an additional anomaly class that needs to be considered is actually involved.

[0036] An advantageous embodiment of the method may include creating the first training data set and the second training data set by randomly selecting (e.g., by bagging) data sets from the overall training data set, respectively. This may also narrow the trend for considering or not considering certain anomalies, and thus make the method more robust against incorrect interpretations.

[0037] A further advantageous embodiment of the method may also include retraining the first machine learning model with the added training data set to generate a third machine learning model. In this case, the parameters of the first machine learning model may be used as starting values ​​for training. As an alternative, the machine learning model may also be completely restarted to create a trend-free machine learning model.

[0038] According to a particular embodiment of the method, the classification can be performed by a classifier selected from the group consisting of: neural networks (including deep neural networks), random forests, logistic regression (usually atypical), support vector machines (SVM) and Gaussian regression. In principle, all known classifiers can be used, from which a person skilled in the art selects a suitable classifier or classifier system depending on the anomaly and the object to be studied.

[0039] According to a further useful embodiment of the method, the data set can be derived from an image recording method. In this case, the image recording method can be based on the following: electron microscopy (particularly also using other charged particles), fluorescence microscopy, light microscopy, optical coherence tomography systems, interferometers, spectrometers, surgical microscopes and computer tomography systems. A person skilled in the art can use comparable image recording methods to perform the method.

[0040] According to a feasible embodiment, the method may further include: receiving a selection signal for anomalies, for example from a visual inspector, wherein the selection signal amplifies the difference value so that the difference value is above the first predefined threshold or above the second predefined novelty threshold value. Using this technique based on overwriting the determined difference value, an "uncertain candidate" can be changed to a "determined candidate", and thereby the annotation can be supported by an expert. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] It is noted that exemplary embodiments of the present invention may be described with reference to different implementation categories. In particular, some exemplary embodiments are described with reference to methods, while other exemplary embodiments may be described in the context of corresponding devices. In any case, it is possible for a person skilled in the art to recognize and combine possible combinations of features of the method and possible combinations of features of the corresponding system based on the above and following descriptions, if not otherwise indicated.

[0042] The aspects already described above and further aspects of the invention become apparent in particular from the exemplary embodiments already described and from further, specific developments described with reference to the attached drawings.

[0043] Preferred exemplary embodiments of the present invention are described by way of example and with reference to the following drawings:

[0044] Figure 1 A block diagram of an exemplary embodiment of a computer-implemented method of the present invention for expanding a training data set for machine learning is presented.

[0045] Figure 2 Block diagram showing a sequence of conventional training sequences for a classifier.

[0046] Figure 3 A block diagram showing a flowchart closer to an implementation of the proposed method is shown.

[0047] Figure 4 A block diagram of an exemplary embodiment of a workflow system is presented.

[0048] Figure 5 A block diagram of a computer system having an additional workflow system is shown. DETAILED DESCRIPTION

[0049] In the context of this description, conventions, terms and / or expressions shall be understood as follows:

[0050] The term "machine learning" is a basic term or basic function in artificial intelligence, in which, for example, a computer system is given the ability to "learn" using static methods. By way of example, in this case, certain behavioral patterns within a specific task are optimized. The methods used give the machine learning system the ability to analyze data without explicit program planning for this purpose. As an example of a machine learning system, a CNN (convolutional neural network) is, for example, a typical network of nodes that act as artificial neurons and artificial links between these artificial neurons, wherein parameters such as the weight parameters of the links can be assigned to the artificial links. When training the neural network, the weight parameters of the links are automatically adjusted based on the input signal to produce the desired results. In the case of supervised learning, metadata (annotations) are added to the images (usually input data) provided as input values ​​(training data) to show the expected output values. In the case of unsupervised learning, such annotations are not necessary. It is generally believed that the mapping of input data to output data is learned.

[0051] In this regard, recurrent neural networks (RNNs) should also be mentioned, which also constitute a type of deep neural network in which weight adjustments are performed recursively, making it possible to generate structured predictions about input data of variable size. Such RNNs are usually used for sequential input data. In this case, in the same way as CNNs, in addition to the forward weight adjustment, a backpropagation function is used. RNNs can also be used for image analysis.

[0052] The term "training dataset" in connection with this document essentially describes image data that can be obtained from one or more anomalies in the object to be investigated (eg, a workpiece) for which annotations are already present, which can be used using the method.

[0053] The term "extending the training dataset" describes the process of extending an existing training dataset with new image data having corresponding classifications, which new image data can then be used for further or extended training of the classifier.

[0054] In this context, the term "object under investigation" usually describes the artifact that is to be quality checked.

[0055] The term "anomaly" in principle describes an undesired abnormality in the volume or surface of the object to be investigated. This may be, for example, a void, an inclusion, a core break, a wall displacement, or another anomaly in or on the workpiece.

[0056] The term “classification” describes the process of assigning detected anomalies to anomaly classes by a classifier operating on the principles of machine learning.

[0057] The term "classifier" (also called classifier system in the context of machine learning) describes a machine learning system that, by being trained with training data, is endowed with the ability to assign input data (here in particular image data of anomalies of the object to be studied) to specific classes (in particular anomalies).

[0058] In this context it should also be noted that classifiers usually classify into a predefined number of classes. Typically this is achieved by a classification value of the input data determined for each class and a WTA (winner takes all) filter that selects the class with the highest classification value as the classified class. In classifiers, the difference from the 100% classification value is usually used as a quality parameter for the classification or the probability of the correctness of the classification.

[0059] In the context of this document, the term "difference value" describes a value determined by the method proposed herein or a corresponding workflow system, which specifies the size of the difference between the detected (i.e., identified) anomaly and 100% classification. In this case, the highest classification value (see above) can be used as an indicator; however, other classification values ​​of other classes can also be used in addition. In the simplest case, it will therefore be possible to simply apply the classification probability parameter of the classifier. This difference value is applied in the decision as to whether the identified anomaly is assigned to a specific class that is already known in a reasonable way, and therefore, the corresponding pixel data is automatically annotated accordingly for the purpose of adding to the training data set. These annotation values ​​of the pixels are then used to add the parameter "annotation value" to the data set consisting of coordinate values ​​and measurement values.

[0060] If a clear classification is not possible, the professional can authorize a new anomaly class or perform further or new rounds of training on the trained classifier.

[0061] The term "training a classifier" refers to the tuning of a machine learning system, for example, by partially repeatedly evaluating several example images, for example in a neural network, after a training phase to assign even unknown images to one or more classes on which the learning system has been trained. The example images are usually annotated (i.e. provided with metadata) in order to produce a desired result based on the input image, for example information about maintenance measures to be taken.

[0062] As an example of a classifier, the term "convolutional neural network" describes a class of artificial neural networks based on feed-forward techniques. They are often used for image analysis using images as input data. The main components of a convolutional neural network in this case are convolutional layers that enable efficient evaluation through parameter sharing (hence the name).

[0063] Mention is also made of the fact that a deep neural network consists of several layers with different functions - for example an input layer, an output layer and one or more insertion layers, e.g. for convolution operations, application of nonlinear functions, dimensionality reduction, normalization functions, etc. These functions can be "performed in software", or special hardware components can take over the calculation of the corresponding function values. Combinations of hardware and software elements are also known.

[0064] The term "novelty detection" describes a mechanism by which an "intelligent" system (e.g., an intelligent organism) is able to classify an arriving sensor pattern as a previously unknown pattern. This principle can also be applied to artificial neural networks or other classifiers. An arriving sensor pattern (e.g., a digital image) can be classified as an image with new content (novelty) if the sensor pattern arriving at the artificial neural network does not generate an output signal with a detection probability above a predefined (or dynamically adjusted) threshold, or generates several output signals with approximately the same detection probability.

[0065] On the other hand, if a new example image is fed to a neural network trained with known example images (and configured, for example, as an autoencoder), and the example image is run through the autoencoder, the autoencoder should be able to reconstruct the new example image again at the output. This is possible because the new example image is compressed to a large extent when running through the autoencoder in order to be subsequently expanded / reconstructed again by the neural network. If the new input image matches (to a large extent) the expanded / reconstructed image, the new example image corresponds to a known pattern or has a high similarity to the content in the training image database. If the comparison of the new example image with the expanded / reconstructed image shows significant differences, i.e. if the reconstruction error is large, the example image is an image with a previously unknown distribution (image content). That is, if the autoencoder cannot map the data distribution, this is an anomaly compared to the known database, which is assumed to be normal.

[0066] A detailed description of the drawings is given below. In this case, it should be understood that all details and information in the figures are shown schematically. First, a block diagram of an exemplary embodiment of a computer-implemented method for expanding a training data set for machine learning according to the present invention is shown. Further exemplary embodiments or exemplary embodiments for corresponding systems are described below:

[0067] Figure 1 A block diagram of an exemplary embodiment of a computer-implemented method 100 of the present invention for expanding a training data set for machine learning is shown. The method 100 includes providing 102 a data set of an object to be studied. In this case, the data set has coordinate values ​​and a measurement value for each coordinate. It can be a 2D (2-dimensional) or 3D data set, with a grayscale value (also commonly referred to as a voxel) for each pixel as a measurement value, or with other color values. There may also be other measurement values ​​for each pixel or group of pixels. The data in the data set are recorded using the above method, and the data may be pre-processed (e.g., normalized).

[0068] The method 100 further comprises identifying 104 anomalies in a partition of the data set corresponding to a subregion of the object to be investigated (also referred to in the literature as a "region of interest"). In one variant, this is thus a partition; in the case of 3D image data, this is a subvolume.

[0069] The method 100 also includes classifying 106 the identified anomalies into a plurality of predefined classification classes by means of a first machine learning model trained with a first training data set. This enables classification of anomalies in the subject under investigation. To typically form the first machine learning model in a classifier for classification, the classifier has been trained with the first training data set prior to use (i.e., prior to classification of the anomaly).

[0070] The method 100 also includes determining 108 a difference value of the anomaly compared to the trained first machine learning model. The difference value is not necessarily the classification probability value of the classifier, but can be derived therefrom - ie, a function thereof.

[0071] The method 100 further comprises: when the difference is above a first predefined threshold, adding 110 data related to the identified anomaly to the first training data set. In this case, the threshold of the difference may be set to a separately predefined quality standard, production process and expected anomaly. Annotation is typically performed per pixel.

[0072] Figure 2 A block diagram of a sequence 200 of a conventional training sequence for a classifier is shown. In this case, potential errors, i.e. anomalies, are first determined 202. These anomalies are annotated according to defect classes 204. In this case, in particular those pixels belonging to the respective anomaly are provided manually and therefore in a time-consuming manner with specific annotation labels for the respective anomaly class. This process is performed interactively by the user. A training data set is thus created. As is known, a large number of example anomalies are required for this purpose.

[0073] In a next step, the classifier is trained with the complete training data set, 206. The trained classifier is then used in a checking process in a quality assurance system, 208. This process is known from the prior art to be time consuming and requires special expertise in assessing anomalies and thus creating training data with corresponding annotations.

[0074] The method proposed herein can use this process to create a basic training data set in order to obtain basic training data, in particular a first training data set.

[0075] Figure 3 A block diagram of a flowchart 300 is shown that is closer to an embodiment of the proposed method. According to the workflow of the proposed method, potential errors, i.e. anomalies, are also first determined in this case, 302. In order to form a first training data set in particular for training a first machine learning model, the corresponding data are usually annotated manually according to defect classes, 304, so that they can be trained later with an increased amount of training data, 306. This classifier trained in this way can also be used in the normal inspection process, 308.

[0076] However, if then during the ongoing inspection it is found that the difference values ​​of the classifications are too large compared to the first predefined threshold (i.e. simply greater than the threshold) based on the probability of classification correctness or the corresponding quality parameter of the classification, then the training data set can be added, in which case a new anomaly class is then considered. At the same time, pixels belonging to the identified anomalies that cannot be clearly classified (i.e. the difference values ​​are less than the predefined threshold) can be automatically annotated with new annotation tags belonging to the new anomaly class.

[0077] Retraining the classifier with the extended training data set then also includes learning the additional classification classes, i.e., the machine learning model of the classifier is also trained with the added training data set for the additional classification classes.

[0078] As described above, the degree of certainty (i.e., threshold) is determined 310 based on predefined parameters of the input data used (typically image data), the material forming the object to be investigated, and / or the expected anomalies or defects. Additional annotation 312 by the degree of certainty of the selected additional data and retraining 306 using the now increased data set have also been explained above.

[0079] This degree can then be used differently in the sequence of inspection processes: (1) In order to make the data set more targeted, only data related to subvolumes with a high estimated value for the newly acquired information is stored. This saves storage space and a large number of relevant subvolumes can be stored in a memory of a predefined size. Then, only those subvolumes that provide particularly useful information are considered for annotation, so that the classifier is expected to improve in subsequent training.

[0080] A high estimated information content, i.e., a high difference value (e.g., due to high uncertainty about the classification), simultaneously indicates that the associated subvolume may contain previously unknown defects. Therefore, this should also be taken into account when extending the training dataset. If faulty images occur frequently, the set of error classes in the defect catalog can be extended with new classes. That is, it is not necessary to introduce new classes every time an anomaly with a difference value larger than a threshold appears for the first time. In this case, the model parameters of the previously trained model can be used to initialize the new model. This can significantly reduce the training time and the amount of additional training data required.

[0081] When new data is inspected, the degree can be immediately evaluated for each local effect so that a decision can be made directly during the inspection process whether it should be submitted to an expert for subsequent inspection (stream-based setting). As an alternative, a sufficient number of local defects (i.e., anomalies) can be collected first and then selected from these for annotation by evaluating the degree of information in the most relevant examples (pool-based setting).

[0082] To prevent irrelevant data points from being selected, factors may be additionally added to the degree of information: for example, a factor for estimating whether visually similar examples appeared when new data was collected (a novel cluster), or whether it is an outlier (e.g., caused by a recording error that occurred in the short term and resulted in an image with a completely different appearance).

[0083] In order to avoid an increase in the number of rejections by visual inspectors, the information degree should be adapted so that similar examples with lower information values ​​are rated later.

[0084] Since the selected examples are annotated by multiple people (multiple annotations), the impact of incorrect annotations by non-application experts can be reduced very easily. As an alternative, the errors caused by the visual inspectors can also be estimated already in the information level, so that different impacts of the annotations are presented to different visual inspectors. This can be done regardless of the reliability of the respective visual inspectors.

[0085] To ensure that the combination of several examples also provides useful information, a degree of diversity can be added to the selection. This can be performed by directly assessing the similarity of the examples, or by assessing the variance of the resulting classifier.

[0086] For the sake of completeness, Figure 4 A block diagram of an exemplary embodiment of a workflow system 400 is shown. The workflow system 400 has a receiving unit 402, which is designed to receive a data set of an object to be studied. In this case, the data set has coordinate values ​​and a measurement value for each coordinate.

[0087] The workflow system 400 also has a detection module 404 and a classification system 406, wherein the detection module is designed to identify anomalies in a partition in the data set corresponding to a sub-region of the object to be studied, and the classification system is designed to classify the anomalies into multiple predefined classification classes by using a first machine learning model trained with a first training data set.

[0088] The difference determination module 408 as part of the workflow system 400 is designed to determine the difference of the anomaly compared to the trained first machine learning model. Finally, the workflow system 400 also has an adding unit 410, which is designed to add data related to the identified anomaly to the training data set when the difference is above a first predefined threshold.

[0089] Therefore, automatic annotation and extension of training datasets as well as insertion of potential new anomaly classes can be elegantly achieved.

[0090] In summary, it can therefore be said that in order to create the solution proposed here, information content measures known from the field of active learning can be used. This basically includes two methodological classes: exploratory methods and mining methods.

[0091] Exploratory methods ignore already trained classifier models and try to cover the entire feature space as much as possible using as few data points to be selected, i.e. to generate a representative, variance-rich data set as quickly as possible. It is therefore assumed that a lot of information is present in those new data which are largest and similar to the known data. With regard to potential problems here, these methods will therefore select defects for annotation that were not previously known in terms of their visual appearance. The advantage of these methods is the option of quickly selecting previously unknown defect classes (anomaly classes) for annotation. There are disadvantages with regard to sensitivity to outliers caused, for example, by interfering processes during data recording. Measures that explicitly address the appearance of new, unknown data can be applied directly from the field of "novelty detection".

[0092] Regarding mining methods, it should be assumed that the classifier has already achieved practical performance in the problem being solved (warm start), and it is not necessary to train the machine learning model from scratch (cold start). Therefore, the degree of information that requires an already trained classifier can be successfully used.

[0093] Mining methods incorporate a previously trained classifier into the assessment of the information content of new examples, i.e., these methods "mine" the classifier. The goal then is to achieve a rapid reduction in classification error. Depending on the amount of available training data and the available computational time, the expected error reduction can be assessed directly (e.g., estimated error reduction or estimated entropy minimization), slightly approximated (e.g., using EMOC), or replaced by a heuristically motivated approach.

[0094] Uncertainty-based sampling is particularly able to take advantage of heuristic methods, meaning that any example has high information content where the current classifier has the largest uncertainty level (e.g., SVM margin, k-nearest neighbors, 1-vs-2 and entropy, GP mean).

[0095] Similarly, one can easily use example selections that had a large influence on a previous classifier in new training, which is called expected model change (e.g., for SVM or random forests).

[0096] In the context of the present application, applying uncertainty sampling is particularly relevant, since the degree can be computed quickly, thereby enabling delay-free interaction between the data recording system and the data annotation experts.

[0097] Information content MC-Dropout and OpenMax are particularly suitable for use with neural networks or deep learning.

[0098] Figure 5A block diagram of a computer system that may have at least a part of a maintenance supervision system is shown.In principle, embodiments of the concepts presented herein may be used with nearly any type of computer, regardless of the platform used therein for storing and / or executing program code. Figure 5 By way of example, a computer system 500 suitable for executing program code according to the method presented here is shown. Computer systems already present in microscope systems can also be used as computer systems for executing the concepts presented here, possibly with corresponding expansions.

[0099] Computer system 500 has several general functions. In this case, the computer system can be a tablet computer, a laptop / notebook computer, another portable or mobile electronic device, a microprocessor system, a microprocessor-based system, a smart phone, or a computer system with special functions in a special configuration. Computer system 500 can be configured to execute computer system executable instructions, such as program modules, which can be executed to implement the functions of the concepts presented herein. To this end, program modules can include routines, programs, objects, components, logic, data structures, etc., to implement specific tasks or specific abstract data types.

[0100] The components of the computer system may have the following: one or more processors or processing units 502, a storage system 504, and a bus system 506 that connects various system components including the storage system 504 to the processor 502. The computer system 500 typically has several volatile or non-volatile storage media that can be accessed by the computer system 500. The storage system 504 can store data and / or instructions (commands) of the storage media in a volatile form (for example, in RAM (random access memory) 508) for execution by the processor 502. These data and instructions perform one or more functions or steps of the concepts presented herein. Other components of the storage system 504 may be a permanent memory (ROM) 510 and a long-term storage 512, in which program modules and data (reference numeral 516) may be stored.

[0101] The computer system has a number of dedicated devices for communication (keyboard 518, mouse / pointing device (not shown), screen 520, etc.). These dedicated devices can also be combined in a touch-sensitive display. A separately provided I / O controller 514 ensures frictionless data exchange with external devices. A network adapter 522 can be used to communicate via a local area network or a global network (LAN, WAN, for example via the Internet). Other components of the computer system 500 can access the network adapter via the bus system 506. In this case, although not shown, it should be understood that other devices can also be connected to the computer system 500.

[0102] At least a portion of the workflow system 400 (see Figure 4 ) can also be connected to the bus system 506. The digital image data from the image sensor (not shown) can also be processed by a separate pre-processing system (not shown). This can support the provision of a data set of the object to be studied.

[0103] The description of various exemplary embodiments of the present invention has been given for the purpose of improving understanding, but it is not intended to limit the inventive concept directly to these exemplary embodiments. Those skilled in the art will develop further modifications and variations on their own. The terms used herein are selected to best describe the basic principles of the exemplary embodiments and to make them easily accessible to those skilled in the art.

[0104] The principles proposed herein may be embodied as systems, methods, combinations thereof, and / or computer program products. In this case, the computer program product may have one (or more) computer-readable storage media containing computer-readable program instructions to prompt a processor or control system to perform various aspects of the present invention.

[0105] Electronic, magnetic, optical, electromagnetic or infrared media or semiconductor systems are used as forwarding media; for example, SSD (solid-state device / drive as solid-state storage), RAM (random access memory) and / or ROM (read-only memory), EEPROM (electrically erasable ROM) or any combination thereof. Propagated electromagnetic waves, electromagnetic waves in waveguides or other transmission media (for example, light pulses in optical cables) or electrical signals transmitted in wires are also considered forwarding media.

[0106] The computer readable storage medium may be an embodiment device that retains or stores instructions for use by an instruction execution device. The computer readable program instructions described herein may also be downloaded from a service provider to a corresponding computer system via a cable-based connection or a mobile radio network, for example as a (smartphone) application.

[0107] The computer-readable program instructions for performing the operation of the present invention described herein may be machine-dependent or machine-independent instructions, microcode, firmware, state definition data, or any source code or object code such as written in C++, Java, etc. or a conventional program programming language (e.g., programming language "C" or similar programming languages). These computer-readable program instructions may be executed completely by a computer system. In some exemplary embodiments, it may also be an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), which executes the computer-readable program instructions by using the state information of the computer-readable program instructions, so as to configure or customize the electronic circuit according to various aspects of the present invention.

[0108] The present invention proposed herein is further illustrated by reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to exemplary embodiments of the present invention.It should be noted that, in fact, any frame of the flow chart and / or block diagram can be designed as a computer-readable program instruction.

[0109] The computer-readable program instructions may be made available to a general-purpose computer, a special-purpose computer, or a data processing system that can be programmed in another manner to create a machine such that the instructions executed by a processor or computer or other programmable data processing device can generate means for implementing the functions or processes illustrated in the flowchart and / or block diagram. These computer-readable program instructions may therefore also be stored on a computer-readable storage medium.

[0110] In this sense, any box in the flowchart or block diagram shown may represent a module, a segment or each part of an instruction, which represents several executable instructions for implementing a specific logical function. In some exemplary embodiments, the functions shown in each box may be executed in another order or in parallel.

[0111] The structures, materials, sequences and equivalents of all methods and / or steps with associated functions shown in the attached technical solutions are intended to apply all structures, materials or sequences expressed in the technical solutions.

[0112] Reference numerals

[0113] 100 Methods

[0114] 102 100 Methods and Steps

[0115] 104 100 Methods and Steps

[0116] 106 100 Method Steps

[0117] 108 100 Methods and Steps

[0118] 110 100 Method Steps

[0119] 200 Methods for annotating training data and using classification

[0120] 202, ... 208 200 Methods and Steps

[0121] 300 Detailed method compared with method 100

[0122] 302, ...312 300 Methods and Steps

[0123] 400 Workflow System

[0124] 402 Receiving Unit

[0125] 404 Detection Module

[0126] 406 Classification System, Classifier

[0127] 408 Difference determination module

[0128] 410 Add Unit

[0129] 500 Computer Systems

[0130] 502 processor

[0131] 504 Storage System

[0132] 506 bus system

[0133] 508 RAM

[0134] 510 ROM

[0135] 512 Long-term storage

[0136] 514 I / O Controller

[0137] 516 Program modules and data

[0138] 518 Keyboard

[0139] 520 screen

[0140] 522 network adapter.

Claims

1. A computer-implemented method (100) for expanding a training data set for machine learning, the method comprising: - providing a data set of the object to be investigated, wherein the data set has coordinate values ​​and a measured value for each coordinate, and wherein the data set is derived from an image recording method using a computed tomography system, - identifying anomalies in a partition of the data set corresponding to a sub-region of the object to be studied, - classifying the anomaly into a plurality of predefined classification classes using a first machine learning model trained with a first training dataset, - determining a difference value for the anomaly as a difference between a degree of novelty determined by the first machine learning model and a degree of novelty determined by a second machine learning model that has been trained using a second training data set, - when the difference is above a first predefined threshold, adding data related to the identified anomaly to the first training data set, and - Creating the first training data set and the second training data set by randomly selecting data sets from the overall training data set, respectively.

2. The method according to claim 1, wherein: The anomaly is a large number of anomalies, and the adding comprises: - when the difference is above a second predefined threshold, adding data related to the selected identified anomaly to the first training data set.

3. The method according to claim 1, wherein: The difference is determined by a novelty degree method for the selected anomaly selected from the group consisting of: novelty detection, estimation error reduction, estimation entropy minimization, expected model output change, MC-dropout, OpenMAX, SVM margin, neural network, ratio of highest classification probability to second highest classification probability, classification probability entropy, GP variance, variance of individual classification probabilities from aggregated classifiers, variance of classification probabilities in the presence of interference on the input signal.

4. The method according to claim 2, wherein: The difference is determined by a novelty degree method for the selected anomaly selected from the group consisting of: novelty detection, estimation error reduction, estimation entropy minimization, expected model output change, MC-dropout, OpenMAX, SVM margin, neural network, ratio of highest classification probability to second highest classification probability, classification probability entropy, GP variance, variance of individual classification probabilities from aggregated classifiers, variance of classification probabilities in the presence of interference on the input signal.

5. The method according to claim 4, wherein: The combination of these novelty degree methods is determined using a method selected from the group consisting of: multi-armed bandit formulation, success-driven selection among multiple criteria, linear combination based on reward functions, reinforcement learning.

6. The method according to claim 1, further comprising - Retraining the first machine learning model with the added training data set to generate a third machine learning model, wherein: The parameters of this first machine learning model are used as starting values.

7. The method according to claim 2, further comprising - Retraining the first machine learning model with the added training data set to generate a third machine learning model, wherein: The parameters of this first machine learning model are used as starting values.

8. The method according to claim 4, further comprising - Retraining the first machine learning model with the added training data set to generate a third machine learning model, wherein: The parameters of this first machine learning model are used as starting values.

9. The method according to claim 5, further comprising - Retraining the first machine learning model with the added training data set to generate a third machine learning model, wherein: The parameters of this first machine learning model are used as starting values.

10. The method according to claim 1 or 2, wherein: The classification is performed by a classifier selected from the group consisting of: neural network, random forest, logistic regression, support vector machine, and Gaussian regression.

11. The method according to claim 3 or 4, wherein: The classification is performed by a classifier selected from the group consisting of: neural network, random forest, logistic regression, support vector machine, and Gaussian regression.

12. The method according to claim 5, wherein: The classification is performed by a classifier selected from the group consisting of: neural network, random forest, logistic regression, support vector machine, and Gaussian regression.

13. The method according to claim 6 or 7, wherein: The classification is performed by a classifier selected from the group consisting of: neural network, random forest, logistic regression, support vector machine, and Gaussian regression.

14. The method according to claim 2, further comprising - receiving a selection signal for an anomaly, wherein, The selection signal amplifies the novelty level so that the novelty level is higher than the first predefined threshold or higher than the second predefined threshold.

15. The method according to claim 4, further comprising - receiving a selection signal for an anomaly, wherein, The selection signal amplifies the novelty level so that the novelty level is higher than the first predefined threshold or higher than the second predefined threshold.

16. The method according to claim 5, further comprising - receiving a selection signal for an anomaly, wherein, The selection signal amplifies the novelty level so that the novelty level is higher than the first predefined threshold or higher than the second predefined threshold.

17. The method according to claim 7, further comprising - receiving a selection signal for an anomaly, wherein, The selection signal amplifies the novelty level so that the novelty level is higher than the first predefined threshold or higher than the second predefined threshold.

18. The method according to claim 8, further comprising - receiving a selection signal for an anomaly, wherein, The selection signal amplifies the novelty level so that the novelty level is higher than the first predefined threshold or higher than the second predefined threshold.

19. A workflow system (400) for expanding a training data set for machine learning, the workflow system (400) having - a receiving unit (402), which is designed to receive a data set of an object to be studied, wherein: The data set has coordinate values ​​and a measurement value for each coordinate, and wherein the data set is derived from an image recording method using a computed tomography system, a detection module (404) designed to identify anomalies in a partition of the data set corresponding to a sub-region of the object to be studied, - a classification system (406) designed to classify the anomaly into a plurality of predefined classification classes by means of a first machine learning model trained with a first training data set, a difference determination module (408) configured to determine a difference value for the anomaly as a difference between a degree of novelty determined by the first machine learning model and a degree of novelty determined by a second machine learning model that has been trained using a second training data set, - an adding unit (410) designed to add data related to the identified anomaly to the first training data set when the difference is above a first predefined threshold, and - a providing module, the providing module being designed to create the first training data set and the second training data set by respectively randomly selecting data sets from the overall training data set.

Images