Methods and apparatus for creating systems that automatically generate machine learning systems

Abstract

A computer-implemented method for creating a system suitable for automatically creating computer-based vision machine learning systems, the method comprising the following steps: providing pre-given hyperparameters; determining optimal parameterizations of the hyperparameters for multiple different training datasets using BOHB (an abbreviation for Bayesian Optimization (BO) and Superband (HB)); evaluating all optimal parameterizations for all training datasets using a normalization metric; creating a matrix having an evaluated normalization metric for each parameterization and each training dataset; determining meta-features for each training dataset; and optimizing a decision tree based on the meta-features and the matrix such that the decision tree outputs which of the optimal parameterizations determined using BOHB is the appropriate parameterization for a given meta-feature.

Description

Technical Field

[0001] This invention relates to a method for creating a system suitable for automatically creating computer-based vision-based machine learning systems, and also to a corresponding computer program and machine-readable storage medium. Background Technology

[0002] A current challenge in machine learning is that the hyperparameters of machine learning algorithms must be reconfigured for each training dataset, based on expert assumptions and experience. Without such configuration, the learning algorithm may converge to a non-optimal solution or even fail to find a solution. This is highly disadvantageous because optimal hyperparameter parameterization is rarely achieved manually. Consequently, machine learning systems that learn using hyperparameters suffer significant performance losses.

[0003] Several approaches have been developed to overcome these shortcomings and, for example, to use machine learning methods to determine the optimal hyperparameters for a given training dataset. For instance, “BOHB: Robusta efficient hyperparameter optimization at scale” by Falkner, Stefan, Aaron Klein, and Frank Hutter, arXiv preprint arXiv: 1807.01774 (2018), is available online at: https: / / arxiv.org / abs / 1807.01774.

[0004] However, the drawback of these schemes is that the hyperparameterizations they discover can only be used conditionally, and are not optimal or reliable for similar datasets, such as datasets with different numbers of categories or containing images from similar domains or similar classification problems.

[0005] Advantages of the present invention

[0006] In contrast, the advantage of this invention lies in providing a method for automatically and optimally parameterizing machine learning algorithms, independent of domain, and an associated machine learning system. Therefore, machine learning systems can be automatically trained using this invention, wherein the learning algorithm can be reliably applied to a large number of different datasets and achieves optimal results, for example, regardless of the object category and / or the number of training images or training videos. Summary of the Invention

[0007] In a first aspect, the present invention relates to a method for creating a computer-implemented system suitable for automatically creating machine learning systems for computer vision (CV).

[0008] Computer-based vision can be understood as machine learning systems configured to process and analyze images, videos, etc., recorded by any type of camera in the most diverse ways. For example, this could be image classification, object detection, or semantic segmentation.

[0009] The method includes the following steps:

[0010] Provide pre-given hyperparameters, specifically defining the range of values ​​for each pre-given hyperparameter. These hyperparameters can be very different parameters and are typically parameterized to the optimization algorithm, particularly the training algorithm, or their values ​​can be assigned to one optimization algorithm from several very different optimization algorithms. The hyperparameters include at least one first parameter characterizing which optimization method is used. The optimization method can be a stochastic optimizer, such as Adam, AdamW, or Nesterov gradient acceleration.

[0011] The hyperparameters also include a second parameter characterizing the type of the machine learning system, and in particular, which function approximators the machine learning system uses. For example, the following types can be used: (preferably pre-trained) EfficientNet or simple classifiers such as SVM, random forest, deep neural networks, or logistic regression.

[0012] Subsequently, BOHB is used to determine the optimal parameterization of the hyperparameters for each of several different training datasets used for computer-based vision. These datasets, referred to as meta-training datasets, are characterized by including input variables with assigned labels. Each input variable can be a 4D tensor (time / column / row / channel). The labels are preferably vectors representing categories or semantic segments through binary values. The datasets are preferably complementary to each other and particularly preferably use publicly accessible datasets such as Chucky, Hammer, Munster, Caltechbirds2010, CIFAR100, CIFAR10, Colonectal Histology, and Eurosat. Complementarity here can be understood as datasets having very different numbers of categories and / or containing images and videos, etc.

[0013] Then, a normalization metric is used to evaluate all optimal parameterizations for all training datasets across the multiple different training datasets. The normalization metric can be, for example, classification accuracy, runtime, or a normalized cost function.

[0014] Then, a matrix is ​​created containing the evaluated, normalized metric as entries for each parameterization and each training dataset.

[0015] Then, meta-features are determined for each training dataset, whereby these meta-features characterize at least the following properties of the training dataset: image resolution, number of classes, number of training data points / test data points, and number of video frames. Note that for images, the meta-feature "number of video frames" can be set to a value of 1.

[0016] Finally, the decision tree is optimized, outputting which of the optimal parameterizations determined by BOHB is suitable for the given meta-feature based on the meta-feature and the matrix. The decision tree is optimized, or its parameters are set, such that the decision tree determines, based on the provided meta-feature and the matrix, which of the parameterizations determined by BOHB is suitable for the current meta-feature. It is important to note that the decision tree is a model selection mechanism.

[0017] The inventors have discovered that the combination of meta-learning and hyperparameter optimization leads to highly domain-independent learning. Furthermore, this method is capable of handling above-average amounts of meta-training datasets and extracting parameterizations suitable for given meta-features from them with minimal overhead (via decision trees). The advantage of decision trees is their speed and low data requirement. Therefore, the meta-learner is relatively small and can be both trained and run in seconds.

[0018] A system suitable for creating the machine learning system can then be initialized. This system then includes the decision tree, and the system initializes the machine learning system and an optimization algorithm for teaching the machine learning system based on the output of the decision tree. The system can then use the optimization algorithm to teach the machine learning system.

[0019] It is suggested that AutoFolio be used to optimize the parameters of the decision tree. AutoFolio is an algorithm for training a selection model that chooses a suitable optimization algorithm and its optimal configuration, and is described by the authors M. Lindauer, H. Hoos, F. Hutter, and T. Schaub in their publication AutoFolio: An Automatically Configured Selector, Journal of Artificial Intelligence 53 (2015): 745-778, which is available online at: http: / / ml.informatik.uni-freiburg.de / papers / 17-IJCAI-AutoFolio.pdf. This is because it has been found that AutoFolio can be used to apply decision trees particularly effectively and usefully for computer vision-based machine learning systems.

[0020] Furthermore, it is recommended to determine the average of the normalization metric across multiple training datasets, and particularly for all optimal hyperparameterizations, wherein the following parameterization from the hyperparameterizations determined by BOHB is selected, whose normalization metric is closest to the average, wherein the evaluated normalization metric for all training datasets is added to the matrix for this configuration. Additionally or alternatively, the following parameterization from the hyperparameterizations determined by BOHB may be selected, which, on average, has the highest normalization metric improvement over the average normalization metric across all training datasets.

[0021] The advantage of this process is the addition of a robust parameterization, which, while not expected to perform exceptionally well on the evaluated standardized metrics, still achieves good performance on all meta-training datasets. Therefore, it provides a parameterization for the decision tree if other optimal parameterizations are unsuitable.

[0022] It is also suggested to use a greedy algorithm to determine a subset of meta-features from multiple meta-features. The advantage of selecting a suitable subset of meta-features is that redundant or even negatively impactful meta-features are removed, thus making the selected model more reliable. Here, iterative checks can be performed to see if each use of the selected subset of meta-features worsens the model's decision.

[0023] It is also recommended to provide an additional training dataset after the selection model has been created. This is preferably an unknown training dataset that was not used to create the selection model, i.e., not included in the meta-training dataset. Meta-features are then determined for the additional training dataset, and the selection model is used to determine suitable parameterization based on the meta-features and the matrix. Based on this suitable parameterization, a machine learning system can then be created and taught using the additional training dataset, for example, by means of the system described above.

[0024] Furthermore, it is suggested that the machine learning system be created based on a first parameter of the hyperparameters, and that an optimization algorithm for the machine learning system be selected based on a second parameter of the hyperparameters, and that the optimization algorithm be parameterized accordingly with the selected configuration.

[0025] Alternatively, the output can be optimized again using a hyperparameter optimizer (preferably BOHB) with the aid of a decision tree to obtain a suitable parameterization, and then the machine learning system and / or the optimization algorithm can be parameterized accordingly using that parameterization.

[0026] Alternatively, configurations can be randomly sampled from the configuration set, or an alternative parameterization can always be used that achieves the highest average improvement in the normalization metric compared to the mean of the normalization metric across all training datasets.

[0027] It should be noted that appropriate parameterization does not necessarily require the use of all hyperparameters. This is because hyperparameters may be partially interdependent; for example, not every type of machine learning system requires weight decay.

[0028] In another aspect, the present invention relates to a computer program configured to perform the above-described methods, and also to a machine-readable storage medium having the computer program stored thereon. Attached Figure Description

[0029] Embodiments of the present invention will now be explained in more detail with reference to the accompanying drawings. In the drawings:

[0030] Figure 1 The workflow of an embodiment of the present invention is illustrated schematically;

[0031] Figure 2 A flowchart illustrating an embodiment of the present invention is shown schematically;

[0032] Figure 3 An embodiment for controlling a robot that is at least partially autonomous is illustrated schematically;

[0033] Figure 4 An embodiment for controlling a manufacturing system is illustrated schematically;

[0034] Figure 5 An embodiment for a control access system is illustrated schematically;

[0035] Figure 6 An embodiment for controlling and monitoring a system is illustrated schematically;

[0036] Figure 7 An embodiment for controlling a personal assistant is illustrated schematically;

[0037] Figure 8 An embodiment for controlling a medical imaging system is illustrated schematically;

[0038] Figure 9 The possible structure of the first training device 141 is shown. Detailed Implementation

[0039] Figure 1 The workflow of an embodiment of the present invention is illustrated schematically. First, different (meta)training datasets 10 are provided. Then, a set of hyperparameters is optimized for each different training dataset 10 using a hyperparameter optimizer 11, preferably BOHB. The optimal hyperparameters 12 are then applied to all training datasets 10 and evaluated using a normalization metric. Finally, each optimal hyperparameter 12 and the normalization metric for each training dataset 10 are recorded in matrix 13.

[0040] For each training dataset 10, a set of meta-features 14 are extracted, which preferably explicitly characterize the corresponding training dataset 10. The meta-features 14 and matrix 13 are then provided to a meta-learner (AutoFolio) 15. The meta-learner 15 then creates a decision tree, which is configured to select 16 the hyperparameters 12 that best optimize the current meta-features 14 based on the meta-features 14 and matrix 13.

[0041] Figure 2 A flowchart 20 schematically illustrates an embodiment of the method according to the present invention.

[0042] The method begins at step S21. Here, the optimal hyperparameters 12 are determined for multiple different training datasets 10 using BOHB 11.

[0043] Then, in the subsequent step S22, the determined optimal hyperparameters 12 are applied to all training datasets 10 used. The optimal hyperparameters are then evaluated using a normalization metric.

[0044] Then, in step S23, the evaluation using a standardized metric is entered into matrix 13. Therefore, matrix 13 contains the following entries for each dataset 10 and optimal hyperparameter 12, each entry corresponding to the evaluated standardized metric value of the corresponding dataset and the corresponding optimal hyperparameter.

[0045] Then, in the subsequent step S24, meta-features 14 are determined for the training dataset. The meta-features 14 and matrix 13 are then used by a meta-learner 15 (preferably AutoFolio) to train a decision tree, which is then able to select appropriate hyperparameters 16 from optimal hyperparameters 12 based on the meta-features 14 and matrix 13, the optimal hyperparameters being determined, for example, by means of BOHB.

[0046] After the decision tree has been fully trained, the next step is step S25. Here, for a new, previously unseen training dataset, its meta-features 14 are determined, and these meta-features are then added to the decision tree. The decision tree then determines, based on these input meta-features 14, which of the optimal hyperparameters 12 are best suited for the unseen training dataset. The machine learning system can then be initialized using the decision tree based on the selected hyperparameters, and additionally, an optimization algorithm can also be initialized based on the selected hyperparameters. The optimization algorithm can then teach the initialized machine learning system based on the unseen training data.

[0047] The training data is preferably camera recordings, wherein the machine learning system is taught to classify objects, detect objects, or segment semantics.

[0048] Figure 3 An actuator 10 is shown interacting with the control system 40 in its environment. The environment is detected at preferred regular time intervals in a sensor 30, particularly in an imaging sensor such as a video sensor. The sensor 30 can also be provided by multiple sensors, such as a stereo camera. Other imaging sensors, such as radar, ultrasonic, or lidar, are also conceivable. Thermal imaging cameras are also conceivable. The sensor signals S from the sensor 30—or each sensor signal S in the case of multiple sensors—are transmitted to the control system 40. The control system 40 thus receives the sequence of sensor signals S. From this, the control system 40 determines a control signal A, which is transmitted to the actuator 10.

[0049] The control system 40 receives a sequence of sensor signals S from the sensor 30 in an optional receiving unit, which converts the sequence of sensor signals S into a sequence of input images x (alternatively, the sensor signals S can be used directly as the input images x). For example, the input images x can be fragments of the sensor signals S or undergo further processing. The input images x include individual frames of a video recording. In other words, the input images x are determined based on the sensor signals S. The sequence of input images x is then fed to the machine learning system from step S25, which in this embodiment is an artificial neural network 60.

[0050] Artificial neural network 60 preferably uses parameters To parameterize, parameters It is stored in and provided by the parameter memory P.

[0051] Artificial neural network 60 determines output variables y from input image x. These output variables y may include, in particular, classification and semantic segmentation of the input image x. The output variables y are fed to an optional shaping unit 80, which determines from there a control signal A to be sent to actuator 10 to correspondingly manipulate actuator 10. The output variables y include information about objects detected by sensor 30.

[0052] Actuator 10 receives control signal A, is correspondingly controlled, and executes a corresponding action. Actuator 10 may include (not necessarily structurally integrated) control logic that determines a second control signal from control signal A and then uses the second control signal to control actuator 10.

[0053] In other embodiments, the control system 40 includes a sensor 30. In yet another embodiment, the control system 40 alternatively or additionally includes an actuator 10.

[0054] In other preferred embodiments, the control system 40 includes one or more processors 45 and at least one machine-readable storage medium 46 on which instructions are stored, which, when executed on the processor 45, cause the control system 40 to perform the method according to the invention.

[0055] In an alternative embodiment, a display unit is provided instead of or attached to the actuator 10.

[0056] Figure 2 The diagram illustrates how the control system 40 can be used to control at least partially autonomous robots, here at least partially autonomous motor vehicles 100.

[0057] Sensor 30 may be, for example, a video sensor preferably arranged in a motor vehicle 100.

[0058] The artificial neural network 60 is configured to reliably identify objects from the input image x.

[0059] The actuator 10 preferably arranged in the motor vehicle 100 can be, for example, the brake, drive, or steering system of the motor vehicle 100. Thus, the control signal A can be determined to cause one or more actuators 10 to be operated in such a way that the motor vehicle 100, for example, prevents a collision with an object reliably identified by the artificial neural network 60, particularly when the object is a specific type of object, such as a pedestrian.

[0060] Alternatively, the at least partially autonomous robot can also be another mobile robot (not shown), such as a robot that moves by flying, swimming, diving, or walking. The mobile robot can also be, for example, a at least partially autonomous lawnmower or a at least partially autonomous cleaning robot. In these cases, the control signal A can also be determined to cause the mobile robot's drive and / or steering system to be operated in such a way that the at least partially autonomous robot, for example, prevents collisions with objects identified by the artificial neural network 60.

[0061] Alternatively or additionally, the display unit can be controlled by the control signal A and, for example, the identified safety area can be displayed. For example, in the case where the motor vehicle 100 has a non-automatic steering system, the display unit 10a can also be controlled by the control signal A so that if it is determined that the motor vehicle 100 is about to collide with one of the reliably identified objects, the display unit 10a outputs an optical or acoustic warning signal.

[0062] Figure 4 One embodiment is shown in which the control system 40 operates the manufacturing machine 11 of the manufacturing system 200 by operating the actuator 10 that controls the manufacturing machine 11. The manufacturing machine 11 may be, for example, a machine for stamping, sawing, drilling and / or cutting.

[0063] Therefore, sensor 30 can be, for example, an optical sensor, which detects characteristics of manufactured products 12a and 12b. It is possible that these manufactured products 12a and 12b are movable. The actuator 10 of the manufacturing machine 11 can be manipulated based on the detected distribution of manufactured products 12a and 12b, so that the manufacturing machine 11 correspondingly performs subsequent processing steps for the correct manufactured products among the manufactured products 12a and 12b. It is also possible that by identifying the correct characteristics of the same manufactured products (i.e., no mismatch) among the manufactured products 12a and 12b, the manufacturing machine 11 correspondingly adapts to the same manufacturing steps to process subsequent manufactured products.

[0064] Figure 5One embodiment is shown, in which a control system 40 controls an access system 300. The access system 300 may include physical access controls, such as door 401. A video sensor 30 is configured to detect people. The detected images can be interpreted using an object identification system 60. If multiple people are detected simultaneously, their identities can be determined particularly reliably, for example, by associating these people (i.e., objects) with each other, such as by analyzing their movements. An actuator 10 may be a lock that releases or retains access control based on a control signal A, such as opening or closing door 401. For this purpose, control signal A can be selected based on the interpretation of the object identification system 60, for example, based on the determined person's identity. Logical access control may also be provided instead of physical access control.

[0065] Figure 6 One embodiment is shown, in which the control system 40 is used to control the monitoring system 400. This embodiment is related to... Figure 5 The difference in the illustrated embodiment is that a display unit 10a, controlled by a control system 40, is provided instead of an actuator 10. For example, an artificial neural network 60 can reliably determine the identity of an object recorded by a video sensor 30 to infer, for example, which objects are suspicious based on that identity, and then select a control signal A to make the object highlighted in color by the display unit 10a.

[0066] Figure 7 One embodiment is shown, in which the control system 40 controls the personal assistant 250. The sensor 30 is preferably an optical sensor that receives an image of a gesture from the user 249.

[0067] Based on the signal from sensor 30, control system 40 determines a control signal A for personal assistant 250, for example, by performing gesture recognition via a neural network. The determined control signal A is then transmitted to personal assistant 250, and thus the personal assistant is controlled accordingly. The determined control signal A can be specifically selected such that it corresponds to a guessed desired control made by user 249. The guessed desired control can be determined based on a gesture recognized by artificial neural network 60. Control system 40 can then select control signal A to transmit to personal assistant 250 based on the guessed desired control, and / or select control signal A to transmit to personal assistant 250 corresponding to the guessed desired control.

[0068] The corresponding operation may include, for example, the personal assistant 250 retrieving information from the database and reproducing the information in a manner that the user 249 can read.

[0069] Instead of the personal assistant 250, household appliances (not shown), particularly washing machines, electric stoves, ovens, microwave ovens, or dishwashers, can also be set up and controlled accordingly.

[0070] Figure 8 One embodiment is shown in which a control system 40 controls a medical imaging system 500, such as an MRT device, X-ray device, or ultrasound device. The sensor 30 may be provided, for example, by an imaging sensor, and the control system 40 manipulates the display unit 10a. For example, a neural network 60 may determine whether an area recorded by the imaging sensor is conspicuous, and then select a manipulation signal A such that the area is highlighted in color by the display unit 10a.

[0071] Figure 9 A possible structure for a training device 141 used to train the neural network 60 after step S25 or the decision tree after step S23 is shown. The training device is parameterized using parameters θ provided by the parameter memory P.

[0072] Training device 141 includes a provider 71 that provides an input image e from a training dataset. The input image e is fed to a machine learning system or decision tree 61 to be trained, which thereby determines an output variable a. The output variable a and the input image e are fed to an evaluator 74, which thereby determines a new parameter θ' using an optimization method as described in the corresponding steps S25 / S23. This new parameter is then transferred to a parameter memory P and replaces the parameter θ there.

[0073] The method executed by the training system 141 can be implemented as a computer program stored on a machine-readable storage medium 147 and executed by the processor 148.

[0074] The term "computer" includes any device used to process pre-given computational rules. These computational rules can exist in software, hardware, or a hybrid of both.

Claims

1. A computer-implemented method for creating a system, which is suitable for automatically creating a machine learning system for computer-based vision, comprising the following steps: providing pre-defined hyperparameters, wherein the hyperparameters comprise at least one first parameter characterizing which optimization method is used and a second parameter characterizing which type of machine learning system is; determining, by means of Bayesian optimization and hyperband, BOHB, (11) an optimal parameterization of the hyperparameters for each training dataset of a plurality of different training datasets (10) for computer-based vision; evaluating all optimal parameterizations for all training datasets of the plurality of different training datasets by means of a standardization metric; creating a matrix (13), wherein the matrix (13) has the evaluated standardization metric for each parameterization and each training dataset; determining meta-features (14) of each training dataset, wherein the meta-features (14) characterize at least one of the following properties of the training dataset: image resolution, number of classes, number of training data points / test data points and number of video frames; initializing a system comprising a decision tree; and optimizing the decision tree according to the meta-features (10) and the matrix (13) such that the decision tree outputs which of the optimal parameterizations determined by means of BOHB is the parameterization (16) suitable for a given meta-feature.

2. The method according to claim 1, wherein the parameters of the decision tree are optimized by means of AutoFolio (15).

3. The method according to claim 1 or 2, wherein an average value of the standardization metric is determined over the plurality of different training datasets, wherein the following parameterization of the hyperparameters determined by means of BOHB (11) is selected, which has a standardization metric that is closest to the average value, wherein this parameterization is added to the matrix (13) with the evaluated standardization metric for all training datasets.

4. The method according to claim 1 or 2, wherein an average value of the standardization metric is determined over the plurality of different training datasets, wherein the following parameterization of the hyperparameters determined by means of BOHB (11) is selected, which has the highest improvement of the standardization metric compared to the average value of the standardization metric, on average over all training datasets (10), wherein this parameterization is added to the matrix (13) with the evaluated standardization metric for all training datasets.

5. The method according to claim 1 or 2, wherein a subset of meta-features is determined from the meta-features (14) by means of a greedy algorithm and the decision tree determines a suitable parameterization according to the subset of meta-features and the matrix (13).

6. The method according to claim 1 or 2, wherein a further training dataset is provided, wherein meta-features (14) are determined for the further training dataset, and then using the decision tree to determine a suitable parameterization from the meta-features (14) and the matrix (13), wherein a machine learning system is created based on the suitable parameterization and the machine learning system is taught based on the further training dataset.

7. The method according to claim 6, wherein the machine learning system is created based on the first parameters, and an optimization algorithm for the machine learning system is selected based on the second parameters, and the optimization algorithm is parameterized in correspondence with the selected configuration.

8. The method according to claim 7, wherein the hyperparameters further comprise parameters characterizing a batch size, a number of data points to be used for training, a learning rate, a number of data points that should be used to evaluate the performance of the machine learning system, a ratio of parameters of the machine learning system that remain unchanged during teaching of the machine learning system, and / or a weight decay.

9. A computer program which is set up to perform the method according to any one of claims 1 to 8.

10. A machine-readable storage medium on which the computer program according to claim 9 is stored.

11. A device for creating a system suitable for automatically creating a machine learning system based on computer-based vision, the device being set up to perform the method according to any one of claims 1 to 8.

Images