Apparatus for and method of automating machine learning

Abstract

A device and computer-implemented method comprising: determining, in a representation (O) of relationships between elements, an element representing a first characteristic of a machine learning pipeline (M); determining, in the representation (O), from the element representing the first characteristic, an element representing a second characteristic of the machine learning pipeline (M); outputting an output for the element representing the second characteristic; detecting an input, in particular an input of a user; if the input meets a requirement, determining a parameter of the machine learning pipeline (M) from the element representing the second characteristic, or, else, not determining the parameter of the machine learning pipeline (M) from the element representing the second characteristic.

Description

Technical Field

[0001] This invention relates to the automation of machine learning, and more particularly to the automation of pipeline construction and pipeline configuration. Background Technology

[0002] A machine learning pipeline can transform raw data into conclusions and functional machine learning models.

[0003] Developing a machine learning pipeline is a complex process that requires a deep understanding of the data, as well as a necessary understanding of the domain and the problem to be solved. This necessitates specialized training in information processing and data analysis, particularly in machine learning. Summary of the Invention

[0004] The devices and computer implementations for automating machine learning provide a white-box approach to semi-automated machine learning development supported by semantic technologies, which allows for convenient, scenario-adaptive construction and configuration of machine learning pipelines, even by non-machine learning experts.

[0005] This device and method enable the development of interpretable, scalable, and configurable machine learning pipelines, which can also be used by non-machine learning experts with minimal machine learning training. This is achieved by using semantic techniques to encode formal representations of domain knowledge and machine learning knowledge within the machine learning pipeline.

[0006] Machine learning pipelines can be developed for multiple different datasets to solve several tasks with similarities. These pipelines are highly maintainable and scalable for future scenarios.

[0007] The computer-implemented method includes: determining elements representing a first characteristic of a machine learning pipeline in a representation of relationships between elements; determining elements representing a second characteristic of the machine learning pipeline in the representation based on the elements representing the first characteristic; outputting output for the elements representing the second characteristic; detecting input, particularly user input; and determining parameters of the machine learning pipeline based on the elements representing the second characteristic if the input meets the requirements, or otherwise, not determining the parameters of the machine learning pipeline based on the elements representing the second characteristic.

[0008] The parameters configure the machine learning pipeline. The elements representing the second characteristic make the machine learning pipeline interpretable for non-machine learning expert users. Non-machine learning expert users can configure the machine learning pipeline via the input. No knowledge of the elements representing the first characteristic of the machine learning pipeline is required.

[0009] In one aspect, the method comprises determining whether the element representing the first property and the element representing the second property have a relationship satisfying a condition, in particular the condition being that the element representing the second property is semantically derivable from the element representing the first property in the representation according to the semantics encoded in the representation.

[0010] Outputting the output comprises prompting the user to make a response, in particular to like or dislike the second element or to select the element representing the second property.

[0011] The method preferably comprises determining the relationship with a function for assessing semantic derivability; detecting the response; and modifying at least one parameter of the function according to the response. In this way, the derivability function is updated based on the input.

[0012] The method can comprise determining a link between two elements according to the response; the semantic derivability graph representing the relationship is constructed by storing machine learning templates corresponding to the two elements and the link between them in the semantic derivability graph.

[0013] The method can comprise determining a plurality of elements semantically derivable from the element representing the first property in the representation; determining the element representing the second property from the plurality of elements according to the input. The user can for example select the element representing the second property used to determine the parameter from a list.

[0014] The method can comprise determining another element representing the second property semantically derivable from the element representing the first property in the representation when the input fails to satisfy the requirement. For example, if the user dislikes the second element, the computation of the element used for the parameter can be repeated.

[0015] In one aspect, the method comprises determining the condition according to the input. Thus, the concept of semantic derivability is updated according to the user preferences.

[0016] In one aspect, the method comprises determining the parameter for a part of the machine learning pipeline; and determining the condition according to the function of the part in the machine learning pipeline.

[0017] The method can comprise determining a group of elements according to the input; and determining a plurality of parameters for the representation of the machine learning pipeline according to the group of elements. In this way, a plurality of parameters related to a first element representing a property of the machine learning pipeline can be determined.

[0018] The method can comprise determining a machine learning model according to the representation of the machine learning pipeline.

[0019] The method can comprise providing raw data comprising images; determining the parameters of the machine learning pipeline for an image classifier model; determining the machine learning model in accordance with the parameters; training the image classifier model with at least one image of the raw data. The machine learning model thus trained is or comprises an image classifier model.

[0020] The method can comprise providing an image; classifying the image with the machine learning model.

[0021] An apparatus for determining a machine learning model is adapted to perform the method. BRIEF DESCRIPTION OF DRAWINGS

[0022] Further advantageous embodiments can be derived from the following description and the attached drawings. In the drawings:

[0023] Figure 1 An apparatus for determining a machine learning model is schematically depicted;

[0024] Figure 2 Steps in a method for determining a machine learning model are schematically depicted;

[0025] Figure 3 Aspects of user interaction are schematically depicted. DETAILED DESCRIPTION

[0026] Figure 1 An apparatus 100 for determining a machine learning model 102 is depicted.

[0027] The apparatus 100 comprises a user interface layer 104, a semantic layer 106, a data and mapping layer 108 and a machine learning layer 110. The apparatus 100 is adapted to perform a computer implemented method which will be described in the following.

[0028] The user interface layer 104 provides an interface to the user for using semantic components of the semantic layer 106 and thus of the whole system.

[0029] The user interface layer 104 comprises a first function 112 which is adapted to dynamically visualize information about data annotations 114 and about machine learning models 116.

[0030] The user interface layer 104 can comprise a graphical user interface function for displaying information about current data annotations 114 and about machine learning models 116.

[0031] The first function 112 is adapted to retrieve information from the user about data annotations 114 and / or machine learning models 116 and / or links L between machine learning ontology templates T.

[0032] In an example, the representation of the machine learning pipeline can be selected by a user by selecting it from a plurality of machine learning pipelines 118 displayed on the graphical user interface.

[0033] In an example, the link L between the machine learning ontology templates T can be selected by a user by selecting the machine learning ontology templates T to be linked from a plurality of machine learning templates displayed on the graphical user interface.

[0034] The information about the machine learning model 116 includes a visualization 120.

[0035] The visualization 120 includes feature groups 122. In an example, the feature groups 122 include icons for a first feature group 122-1, a second feature group 122-2, and a third feature group 122-3. The first feature group 122-1 in the example is a group for individual features. The second feature group 122-2 in the example is a group for time series. The third feature group 122-3 in the example is a group for quality indicators.

[0036] The visualization 120 includes a graphical representation of processing algorithms 124. The graphical representation of processing algorithms 124 includes an icon for a first algorithm 124-1, an icon for a second algorithm 124-2, and a graph 124-3. The first algorithm 124-1 in the example is an algorithm for the first feature group 122-1 (e.g., a group for individual features). The second algorithm 124-2 in the example is an algorithm for the second group 122-2 (e.g., a group for time series). The graph 124-3 in the example shows the progression of electrical resistance over time from the origin of a Cartesian coordinate system with a length and with a peak (maximum) value and a drop from the peak to an end value.

[0037] The visualization 120 includes a graphical representation of machine learning algorithms 126. The graphical representation of machine learning algorithms 126 in the example includes an icon 126-1 that depicts aspects of the machine learning algorithm 126.

[0038] The information about the current data annotation 114 includes feature groups 128. In an example, the feature groups 128 include icons for a first feature group 128-1, a second feature group 128-2, and a third feature group 128-3.

[0039] The information about the current data annotation 114 includes domain feature names 130. In an example, the domain feature names 130 include icons for a first domain feature name 130-1, a second domain feature name 130-2, a third domain feature name 130-3, a fourth domain feature name 130-4, and a fifth domain feature name 130-5.

[0040] In an example, the first domain feature name 130-1 represents a status of the data. In an example, the second domain feature name 130-2 represents a characteristic of the data. In an example, the third domain feature name 130-3 represents a type of the data. In an example, the fourth domain feature name 130-4 represents another type of the data. In an example, the fifth domain feature name 130-5 represents a quality of the data.

[0041] In an example, the first domain feature name 130-1 and the second domain feature name 130-2 are mapped to the first feature group 128-1. In an example, the third domain feature name 130-3 and the fourth domain feature name 130-4 are mapped to the second feature group 128-2. In an example, the fifth domain feature name 130-5 is mapped to the third feature group 128-3.

[0042] The information about the current data annotation 114 includes original feature names 132. In an example, the original feature names 132 include icons for a first original feature name 132-1, a second original feature name 132-2, a third original feature name 132-3, a fourth original feature name 132-4, and a fifth original feature name 132-5.

[0043] In an example, the first original feature name 132-1 represents a status code of the data. In an example, the second original feature name 132-2 represents a characteristic of the data. In an example, the third original feature name 132-3 represents a type of the data. In an example, the fourth original feature name 132-4 represents another type of the data. In an example, the fifth original feature name 132-5 represents a quality of the data.

[0044] In an example, the first original feature name 132-1 is mapped to the first domain feature name 130-1. In an example, the second original feature name 132-2 is mapped to the second domain feature name 130-2. In an example, the third original feature name 132-3 is mapped to the third domain feature name 130-3. In an example, the fourth original feature name 132-4 is mapped to the fourth domain feature name 130-4. In an example, the fifth original feature name 132-5 is mapped to the fifth domain feature name 130-5.

[0045] The visualization 120 includes a display T' of a plurality of machine learning ontology templates T that are linkable. A machine learning ontology template T can refer to one of the feature groups, one of the machine learning algorithms, one of the domain feature names, or one of the original feature names, and is represented by one of the icons.

[0046] Figure 1 The arrows connecting these items represent user-provided links L between these elements.

[0047] The semantic layer 106 includes a representation O of relationships for a plurality of elements. More specifically, the representation O includes a first ontology 134, a second ontology 136, and a third ontology 138.

[0048] The first ontology 134 in the example is a domain ontology that encodes domain knowledge in formal representation, including classes and properties of the domain and their relationships.

[0049] The second ontology 136 in the example is a feature group ontology that stores links between domain ontology terms and feature groups in a pre-designed catalog of machine learning pipelines.

[0050] The third ontology 138 in the example is a machine learning pipeline ontology that encodes machine learning knowledge in formal representation, including allowed and default feature groups, suitable feature processing algorithms for feature groups, corresponding feature processing groups, and suitable machine learning modeling algorithms for each feature processing group and their relationships. The third ontology 138 in the example encodes a catalog. The catalog stores in formal representation some successful and reasonably general machine learning pipelines that are pre-designed by machine learning experts. A machine learning pipeline is a pre-designed mapping from a feature group to its feature processing algorithm for the feature group, to its corresponding feature processing group, and to its designated machine learning modeling algorithm for the feature processing group.

[0051] The semantic layer 106 includes a machine learning ontology MLO and a machine learning ontology template T.

[0052] The machine learning ontology MLO encodes machine learning knowledge in formal representation. The machine learning ontology MLO, for example, encodes what feature groups or combinations thereof are allowed or disallowed for processing input data and / or a default feature group for processing input data. The machine learning ontology MLO, for example, encodes what feature processing algorithms are suitable for which feature group. The machine learning ontology MLO, for example, encodes feature processing groups corresponding to feature processing algorithms, machine learning modeling algorithms suitable for feature processing groups, and / or their relationships.

[0053] The machine learning ontology MLO defines, in one aspect, an allowed search space of a semantically available graph.

[0054] The machine learning ontology template T is, in one aspect, an ontology fragment with variables for instantiating the third ontology 138, i.e., the machine learning pipeline ontology.

[0055] The semantic layer 106 includes a dynamic extender, e.g., a dynamic smart reasoner DSR, that takes as input the machine learning pipeline ontology MLO and the machine learning template T and user-provided links L and dynamically extends, configures, and constructs the third ontology 138, i.e., the machine learning pipeline ontology.

[0056] The third ontology 138 (i.e., the machine learning pipeline ontology constructed using the machine learning ontology MLO and the template T) encodes the semantic reachable graph G for a set of elements such as feature groups, feature processing algorithms, feature processing groups, and its designated machine learning modeling algorithm for each feature processing group.

[0057] In one example, the semantic reachable graph G is constructed by linking the machine learning template T using the link L.

[0058] In an example, the dynamic smart reasoner DSR has a two-layer functionality.

[0059] First, the dynamic smart reasoner DSR is configured to take the link L as input and link the machine learning template T to dynamically configure, extend, and construct the semantic reachable graph G.

[0060] In an example, the allowed search space of the semantic reachable graph G is defined based on the machine learning pipeline ontology MLO.

[0061] Second, the dynamic smart reasoner DSR is configured to take, as input, the annotation A of, for example, the input dataset D and compute a number of sets of elements S1,..., Sn in the representation O that are semantically reachable from the link L and can have semantic reachability relationships between each other. An element Si that has a semantic reachability relationship with another element Sj is semantically reachable from the other element Sj.

[0062] The dynamic smart reasoner DSR is configured to dynamically update the semantic reachability relationships in the semantic reachable graph G.

[0063] The semantic layer 106 includes a second functionality SR. In an example, the second functionality SR includes a first reasoner 140, a second reasoner 142, and a third reasoner 144. The second functionality SR can also include an annotator 146.

[0064] The annotator 146 allows a user to annotate raw data with terms from the first ontology 134 (in an example, a domain ontology).

[0065] The first functionality 112 can be adapted to determine, from user input, an annotation A of data. The second functionality SR can be adapted to receive the annotation A. The first functionality 112 can be adapted to determine, from user input, a representation M of a machine learning pipeline. The second functionality SR can be adapted to receive the representation M of the machine learning pipeline.

[0066] The first function 112 can be adapted to the user interaction. More specifically, the first function 112 is adapted to determine and output an output prompting the user to make a response. In one aspect, the first function 112 is adapted to output a prompt displaying the element of the representation O for which the user interaction is initiated. The first function 112 can be adapted to request the user to like or dislike the element. The first function 112 can be adapted to request the user to select one or more of the elements from a list of elements. The first function 112 can be adapted to detect an input by the user in response to the output. The first function 112 can be adapted to determine a result of the user interaction from the input. The first function 112 can be adapted to determine whether the element for which the user interaction is initiated is to be used to determine the result of the machine learning pipeline, indicated. The first function 112 can be adapted to determine a result indicating the element or elements of the list that have been selected by the user from the input. The first function 112 can be adapted to determine a group of elements including the element for which the user interaction is initiated as the result.

[0067] In one aspect, the first function 112 is adapted to determine at least one condition for evaluating the semantic availability of the elements in the representation O.

[0068] In an example, the annotator 146 is adapted to process the annotation A and the elements from the first ontology 134 to determine a first data set 148 including the raw data 150 from the raw database 152 and a first mapping 154 of the raw feature names from the raw data 150 to the domain feature names according to the first ontology 134.

[0069] In an example, the first reasoner 140 is adapted to process the first mapping 154 to determine a second mapping 156 of the domain feature names to the feature groups according to the second ontology 136. The first reasoner 140 can be adapted to automatically generate and configure the second mapping 156 based on the second ontology 136 and the user’s configuration and the result of the user interaction, e.g. the input from the user.

[0070] In an example, the second reasoner 142 is adapted to process the second mapping 156 to determine a third mapping 158 of the feature groups to the processing algorithms to the feature processing groups according to the second ontology 136 and the third ontology 138. The second reasoner 142 can be adapted to automatically generate the third mapping 158 based on the second ontology 136, the catalog, e.g. based on the third ontology 138, the second mapping 156, the representation M of the machine learning pipeline selected by the user and the result of the user interaction, e.g. the input from the user.

[0071] In an example, the second reasoner 142 is adapted to determine a fourth mapping 160 of the domain names to the feature groups to the processing algorithms according to the second mapping 156 and the third mapping 158 and the result of the user interaction, e.g. the input from the user.

[0072] In an example, the second reasoner 142 is adapted to determine a fifth mapping 162 of feature groups to feature processing groups from the third mapping 158 and a result of a user interaction, e.g. input from a user.

[0073] In an example, the third reasoner 144 is adapted to determine a sixth mapping 164 of feature processing groups to machine learning algorithms from the third mapping 158. The third reasoner 144 can be adapted to automatically generate the sixth mapping based on the catalog, e.g. based on the third ontology 138, the representation M of the machine learning pipeline selected by the user, and a result of a user interaction, e.g. input from a user.

[0074] The machine learning layer 110 comprises a first data integrator 166 adapted to determine the raw data 150 from the raw database 152. The machine learning layer 110 comprises a second data integrator 168 adapted to determine the integrated data 170 from the first data set 148. The raw data stored in the raw database 152 is converted by the second data integrator 168 into integrated data 170 suitable for machine learning. The integrated data 170 is provided for a second data set 172 comprising the integrated data 170 and the fourth mapping 160.

[0075] The machine learning layer 110 comprises a first processor 174 adapted to determine features 176 for a third data set 178 from the second data set 172. The third data set 178 comprises the features 176 and the fifth mapping 162.

[0076] The machine learning layer 110 comprises a second processor 180 adapted to determine selected features 182 for a fourth data set 184 from the third data set 178. The fourth data set 184 comprises the selected features 182 and the sixth mapping 164.

[0077] The machine learning layer 110 comprises a third processor 186 adapted to determine the machine learning model 102 from the fourth data set 184.

[0078] The data and mapping layer 108 comprises the raw database 152, the first data set 148, the second data set 172, the third data set 178, the fourth data set 184 and the machine learning model 102.

[0079] In Figure 1 In an example, the second reasoner 142 is adapted to determine a fifth mapping 162 of feature groups to feature processing groups from the third mapping 158 and a result of a user interaction, e.g. input from a user.

[0080] An exemplary method of automatic machine learning pipeline construction and machine learning pipeline configuration is described below. The automated method uses semantics which allow for user interaction during the automated construction of a machine learning pipeline.

[0081] According to the method, the user needs to annotate the data with domain ontology terminology and select a pre-designed machine learning pipeline from a catalog. The user can interact at intermediate steps of a sequence of steps for constructing a representation M of a machine learning pipeline. Everything else is automatic. This is achieved by encoding formal representations of domain knowledge and machine learning knowledge in a catalog of pre-designed machine learning pipelines with semantic techniques.

[0082] In one aspect, the representation M of the machine learning pipeline has parameters. The representation O can include semantics encoding the domain and machine learning knowledge. The semantics include a plurality of elements representing the domain knowledge or the machine learning knowledge and their relationships.

[0083] In one aspect, when the annotation A is provided, a first element in the representation O is determined from the annotation A. A second element in the representation O can then be determined from the first element. In an example, the second element is determined such that the first element and the second element have a relationship satisfying a first condition. The first condition can be that the second element is semantically derivable from the first element in the representation O according to the semantics encoded in the representation O. When the first element and the second element satisfy the first condition, the parameter is determined from the second element. Otherwise, the parameter is not determined from the second element.

[0084] In another aspect, a third element in the representation O can be determined from the second element. In an example, the second element and the third element have a relationship satisfying a second condition. The second condition can be that the third element is semantically derivable from the second element in the representation O. In this aspect, if the second condition is satisfied, the parameter can be determined from the second element and the third element. Otherwise, the parameter can not be determined from the second element and the third element.

[0085] For example, the element S1 can instantiate a first parameter P1 with a group of features occurring in the representation O.

[0086] For example, the element S2 can instantiate a second parameter P2 with one or more processing algorithms occurring in the representation O that are semantically derivable from the group of features of the element S1.

[0087] For example, the element S3 can instantiate a third parameter P3 with one or more groups of processing features occurring in the representation O that are semantically derivable from the one or more processing algorithms of the element S2.

[0088] For example, the element S4 can instantiate a fourth parameter P4 with a processing feature occurring in the representation O that is semantically derivable from the one or more groups of processing features of the element S3.

[0089] For example, element S5 can instantiate a fifth parameter P5 with one or more machine learning algorithms semantically derivable from one or more processing features of element S4 occurring in representation O.

[0090] As the ontology is not graphical, the notion of semantic derivability can vary and depend on the use case and application.

[0091] In one aspect, semantically derivable can be determined based on projecting one or more of the aforementioned ontologies onto a graph structure and computing graph derivability. The graph structure can also account for cohesion of paths between ontology elements. It can account for explicit and implicit relationships between elements of representation O.

[0092] Reference is made below to Figure 2 The method is described. As an example, the method is for a machine learning pipeline having parameters P1,..., Pk. The machine learning pipeline can have a parameter set represented by elements in a representation O.

[0093] The method comprises a step 200 of providing a representation M of a machine learning pipeline having parameters P1,..., Pk.

[0094] Providing the representation M can comprise detecting a user input identifying one of the plurality of representations 118 of the machine learning pipeline, and selecting the representation M of the machine learning pipeline identified in the user input.

[0095] The method comprises a step 202 of providing a representation O. The representation O comprises a plurality of elements and relationships thereof.

[0096] The representation O in the example comprises a first ontology 134, a second ontology 136, and a third ontology 138. Graphs representing these can similarly be used.

[0097] The method comprises a step 204 of providing input data.

[0098] The method comprises a step 206 of providing annotations A of features of the input data, in particular variable names in the input data.

[0099] The method comprises a step 208 of providing a first element and a second element of the representation O. In the example, providing the first element comprises determining an element of the representation O for the annotations A representing features of the input data in the representation O. In the example, variable names of the input data according to the first ontology are determined.

[0100] In the example, providing the second element comprises determining an element of the representation O for the first element representing feature names in the representation O. In the example, variable names in the domain according to the first ontology are determined.

[0101] The second element can be determined from the first element in the representation O such that the first element and the second element have a relationship that satisfies the first condition. In an example, this means that the second element is semantically derivable from the first element in the representation O according to the semantics encoded in the representation O. Multiple second elements that satisfy the first condition can be determined. The method is described for one of these second elements and equally applies to any number of these second elements.

[0102] The method further comprises a step 210 of determining a first data set 148 comprising the input data 150, the first element and the second element.

[0103] The method further comprises a step 212 of determining a third element in the representation O that represents a group of features in the representation O. The first parameter P1 can be determined from the third element.

[0104] Step 212 in an example comprises determining an element of the representation O for the second element, in particular from the second ontology 136, that represents a group of features in the representation O that satisfy a second condition. The method is described for one of these third elements and equally applies to any number of these third elements.

[0105] The third element can be determined from the second element in the representation O such that the third element is semantically derivable from the second element in the representation O.

[0106] The third element can be determined from the second ontology 136.

[0107] The method can comprise determining multiple third elements in the representation O that are semantically derivable from the second element in a relationship between the elements.

[0108] The method optionally comprises a step 213 of a user interaction for the third element or the multiple third elements.

[0109] The user interaction will be described below with reference to Figure 3 The first parameter P1 can be determined from the third element or not from the result of the user interaction for the third element.

[0110] Step 214 is performed when the result of the user interaction indicates that the third element satisfies the first requirement. Otherwise, step 212 is performed. The first requirement can be satisfied if the third element is liked or selected by the user. Otherwise, the first requirement can not be satisfied.

[0111] The second condition can be updated, changed or determined from the result of the user interaction for performing step 212 by the updated notion of semantic derivability.

[0112] The second condition can be updated, changed or determined from a function of a part of the machine learning pipeline in which the first parameter P1 is used.

[0113] The result of the user interaction can indicate a group of elements including a third element, determined from the input received in the user interaction, as described below. The method can include determining a plurality of parameters for the representation M of the machine learning pipeline from the group of elements, including a parameter for the third element.

[0114] The method further includes a step 214 of determining a fourth element in the representation O. Determining the fourth element in the example includes determining an element of the representation O for the third element, representing a processing algorithm that satisfies a third condition. The fourth element can be determined in the representation O from the third element such that the fourth element is semantically derivable from the third element. The fourth element represents a processing algorithm of the group of features represented by the third element. The second parameter P2 can be determined from the fourth element. The method is described for one of these fourth elements, and equally applies to any number of these fourth elements.

[0115] The fourth element can be determined from the third ontology 138.

[0116] The method can include determining a plurality of fourth elements that are semantically derivable from the relationship of the third element in the representation O of the elements.

[0117] The method optionally includes a step 215 of user interaction for the fourth element or the plurality of fourth elements.

[0118] The user interaction will be described below with reference to Figure 3

[0119] The second parameter P2 can be determined from the fourth element or not from the result of the user interaction for the fourth element.

[0120] Step 216 is performed when the result of the user interaction indicates that the fourth element satisfies a second requirement. Otherwise, step 214 is performed.

[0121] The second requirement can be satisfied if the fourth element is liked or selected by the user. Otherwise, the second requirement can not be satisfied.

[0122] The third condition can be updated, changed or determined from the result of the user interaction for performing step 214 by the updated notion of semantic derivability.

[0123] The third condition can be updated, changed or determined from the function of the part in the machine learning pipeline for which the second parameter P2 is used.

[0124] ​The result of the user interaction can indicate a group of elements including a fourth element, the fourth element determined from the input received in the user interaction, as described below. The method can include determining a plurality of parameters for the representation M of the machine learning pipeline from the group of elements, the plurality of parameters including a parameter for the fourth element.

[0125] The method further includes a step 216 of determining a second data set 172 including the second data set, the second element, the third element, and the fourth element.

[0126] The method further includes a step 218 of determining the feature set 176 by processing the second data set 172 with the processing algorithm represented by the fourth element.

[0127] The method further includes a step 220 of determining a fifth element. Determining the fifth element in an example includes determining an element representing O for the third element, which represents the feature set 172 satisfying the fourth condition. In an example, the fifth element is determined such that the fifth element is semantically derivable from the third element in the representation O, and the fifth element is semantically derivable from the fourth element in the representation O. In an example, the fifth element represents one or more processing feature groups.

[0128] The method can include determining a plurality of fifth elements semantically derivable from the third element and the fourth element in the representation O of the relationship between the elements.

[0129] The method optionally includes a step 221 of a user interaction for the fifth element or the plurality of fourth elements.

[0130] The user interaction will be described below with reference to Figure 3 The user interaction.

[0131] The third parameter P3 can be determined from the fifth element or not from the result of the user interaction for the fifth element. The fourth parameter P4 can be instantiated with the processing feature, i.e. with the feature set 176, or not from the result of the user interaction for the fifth element.

[0132] When the result of the user interaction indicates that the fifth element satisfies the third requirement, then step 222 is performed. Otherwise, step 220 is performed.

[0133] The third requirement can be satisfied if the fifth element is liked or selected by the user. Otherwise, the third requirement can not be satisfied.

[0134] The fourth condition can be updated, changed, or determined from the result of the user interaction for performing step 220 with the updated concept of semantic derivability.

[0135] The fourth condition can be updated, changed, or determined from the function of the part in the machine learning pipeline that the third parameter P3 is used for.

[0136] The fourth condition can be updated, changed, or determined based on the functionality of the part of the machine learning pipeline to which the fourth parameter P4 is used.

[0137] The result of a user interaction can indicate a group of elements including a fifth element, which is determined based on the input received during the user interaction, as described below. The method can include determining multiple parameters for a representation M for a machine learning pipeline based on the group of elements, said multiple parameters including parameters for the fifth element.

[0138] The method also includes step 222, which determines a third dataset 178 comprising feature set 176, a third element, and a fifth element.

[0139] The method further includes step 224, which determines a sixth element. Determining the sixth element includes determining, for the fifth element, an element representing the machine learning modeling algorithm that satisfies the fifth condition. In the example, the sixth element is determined such that the sixth element is semantically available from the fifth element in the representation O.

[0140] The method may include determining a plurality of sixth elements that are semantically available from the representation O of the relationships between the fifth elements.

[0141] The method may optionally include step 225 for user interaction with the sixth element or multiple sixth elements.

[0142] The following will refer to Figure 3 Describe user interaction.

[0143] The fifth parameter P5 can be determined based on one or more machine learning algorithms represented by the sixth element, or not based on the results of user interactions used with the sixth element.

[0144] If the result of the user interaction indicates the use of the sixth element, proceed to step 226. Otherwise, proceed to step 224.

[0145] The fifth condition can be updated, changed, or determined based on the results of user interaction, in order to perform step 224 through the updated concept of semantic availability.

[0146] The fifth condition can be updated, changed, or determined based on the functionality of the part of the machine learning pipeline to which the fifth parameter P5 is used.

[0147] The result of the user interaction can indicate a group of elements including a sixth element, which is determined based on the input received during the user interaction, as described below. The method can include determining multiple parameters for a representation M for a machine learning pipeline based on the group of elements, including parameters for the sixth element.

[0148] The method further comprises a step 226 of determining a fourth data set 184 comprising features selected from the set of features 176, the fifth element and the sixth element.

[0149] The method further comprises a step 228 of looking up instructions for processing the data and / or defining data for the fourth data set 184 according to the sixth element.

[0150] In this way, the machine learning pipeline turns raw input data into a conclusion and a functional machine learning model.

[0151] The machine learning pipeline can be applied, for example, to provide an image classifier model. The image classifier model can be used for monitoring and process control based on images taken during a process. The process can be resistance welding. The images can depict the parts being welded and / or the weld.

[0152] To train the image classifier model, a user can annotate training images from the process or a simulation thereof. In the user interaction, the user can select features from the domain feature names that are to be used for image classification. In the user interaction, the user can select a specific processing algorithm for processing or classifying the images. The machine learning modeling algorithm can be determined automatically depending on the user selections. The machine learning modeling algorithm can be executed with parameters according to the representation M of the machine learning pipeline to automatically form a trained machine learning model from the training images.

[0153] For the training, the raw data 150 comprises images. In the training, at least one parameter of the machine learning pipeline M is determined, which is a parameter for the image classifier model. The parameter for the image classifier model can be a hyperparameter or a weight of an artificial neural network. The machine learning model is determined according to the parameter and trained with at least one image of the raw data 150.

[0154] After the training, the images can be classified with the machine learning model thus trained.

[0155] In the above method, the user interacts at intermediate steps to construct the representation M of the machine learning pipeline. One or more parameters for the steps correspond to parts of the machine learning pipeline and are semi-automatically determined. In this context, semi-automatically means that a series of steps is repeated in iterations until the result of the user interaction indicates that the user has selected or liked one or more elements of the corresponding part of the machine learning pipeline.

[0156] In the user interaction, the element determining the first characteristic of the machine learning pipeline M is determined in the representation O of the relationship between the elements. Depending on which step of the series is processed, the element can be the second element, the third element, the fourth element or the fifth element.

[0157] In the user interaction, an element representing a second property of M is determined in O from the element representing the first property. Which element is determined depends on the step of the series being processed. In the corresponding user interaction, a third element is determined from the second element, a fourth element is determined from the second and third elements, a fifth element is determined from the fourth element, or a sixth element is determined from the fifth element.

[0158] The element representing the first property and the element representing the second property have a relationship that satisfies a condition for evaluating semantic availability applicable to the step of the series.

[0159] When a graph is used to evaluate semantic availability, nodes of the graph can represent the elements. The condition can be determined from a property of a first node in the graph of the element representing the first property. Whether the condition is satisfied can be determined from a property of a second node in the graph of the element representing the second property. In an example, the condition is defined by a property of the first node, and the second node is semantically available from the first node if a property of the second node satisfies the condition. For example, a change in the condition is stored in the graph as an updated or new property of the first node or the second node.

[0160] Figure 3 Steps in the user interaction are described for iteration.

[0161] In step 300, the user interaction includes determining an output that prompts a user to make a response.

[0162] The output is, for example, a prompt to display an element for which the user interaction is initiated. The prompt can request that the user like or dislike the element. The prompt can request that the user select the element.

[0163] When the user interaction is initiated for a group of elements, the prompt can include a list of the elements in the group and a request to select one or more of the elements.

[0164] The user interaction includes a step 302 of outputting the output. In an example, the prompt is displayed to the user.

[0165] The user interaction includes a step 304 of detecting an input. The input can be a like or dislike attribute for the element displayed to the user, or a selection of the element from a plurality of elements, such as a list of elements displayed to the user.

[0166] The input can be used to determine a parameter of the machine learning pipeline M from the element if the element is selected, or not to determine the parameter of the machine learning pipeline M from the element in other cases.

[0167] The user interaction can comprise determining the relation by a function for assessing semantic availability. In an example, the function is a function of a part of the machine learning pipeline for which a parameter of the user interaction is used. In this case, the user interaction can comprise detecting the response and modifying at least one parameter of the function in dependence on the response.

[0168] In an example, the function is implemented as follows:

[0169] The dynamic smart reasoner DSR takes as input the link L and the link machine learning template T to dynamically configure, extend and / or construct the semantic availability graph G.

[0170] In an example, the allowed search space of the semantic availability graph G is defined based on the machine learning pipeline ontology MLO.

[0171] The dynamic smart reasoner DSR takes as input the annotation A and computes a number of sets of elements S1,..., Sn in the representation O that are semantically available due to the link L and can have semantic availability relations between each other.

[0172] In an example, the dynamic smart reasoner DSR dynamically updates the third ontology 138, i.e. the machine learning pipeline ontology.

[0173] The function can determine whether a condition is met. For example, if the condition is met, the function can indicate that an element representing a second property is semantically available from an element representing a first property in the representation according to the semantics encoded in the representation, otherwise, it does not indicate this.

[0174] The user interaction comprises a step 306 of determining a result of the user interaction from the input.

[0175] The result can indicate whether the element for which the user interaction was initiated will be used for determining the machine learning pipeline.

[0176] The result can indicate one element or a number of elements of the plurality of elements, in particular the element selected by the user from the input.

[0177] The result can indicate another group of elements comprising the element for which the user interaction was initiated.

Claims

1. A computer-implemented method, characterized in that, In a representation (O) of the relationship between elements, an element representing a first characteristic of the machine learning pipeline is determined; in the representation (O), an element representing a second characteristic of the machine learning pipeline is determined based on the element representing the first characteristic; The output is given for the element representing the second characteristic; the input is detected; if the input meets the requirements, the parameters of the machine learning pipeline are determined based on the element representing the second characteristic; otherwise, the parameters of the machine learning pipeline are not determined based on the element representing the second characteristic. Determine whether the elements representing the first characteristic and the elements representing the second characteristic have a relationship that satisfies the condition that the elements representing the second characteristic are semantically obtainable from the elements representing the first characteristic in the representation (O) according to the semantics encoded in the representation (O). The relation is determined using a function that evaluates semantic availability; this function is a part of a machine learning pipeline whose parameters are used in response to user interactions. The machine learning model is determined based on the representation of the machine learning pipeline. The system provides raw data including images, determines the parameters of the machine learning pipeline for an image classifier model, determines the machine learning model based on the parameters, and trains the image classifier model using at least one image from the raw data.

2. The method according to claim 1, characterized in that, The output includes prompts for the user to respond.

3. The method according to claim 2, characterized in that, The response is either to like or dislike an element representing the second characteristic, or to select an element representing the second characteristic.

4. The method according to any one of claims 2 or 3, characterized in that, The response is detected; and at least one parameter of the function is modified based on the response.

5. The method according to claim 4, characterized in that, The link between the two elements is determined based on the response; the semantically available graph representing the relationship is constructed by storing machine learning templates corresponding to the two elements and the link between them in the semantically available graph.

6. The method according to any one of claims 1 to 3, characterized in that, Determine a plurality of elements that are semantically available in the representation (O) from the elements representing the first characteristic; determine the element representing the second characteristic from the plurality of elements based on the input.

7. The method according to any one of claims 1 to 3, characterized in that, When the input fails to meet the requirement, another element representing the second characteristic is determined that is semantically available in the representation (O) from the element representing the first characteristic.

8. The method according to any one of claims 1 to 3, characterized in that, The conditions are determined based on the input.

9. The method according to any one of claims 1 to 3, characterized in that, The parameters for a portion of the machine learning pipeline are determined; and the conditions are determined based on the functionality of that portion of the machine learning pipeline.

10. The method according to any one of claims 1 to 3, characterized in that, The input is used to determine a group of elements; and the group of elements is used to determine multiple parameters for the representation of the machine learning pipeline.

11. The method according to claim 1, characterized in that, Provide an image; classify the image using the machine learning model.

12. An apparatus for determining a machine learning model, characterized in that, The device is adapted to perform the method according to any one of claims 1 to 11.

13. A computer program product, characterized in that, The computer program product includes computer-readable instructions that, when executed on a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 11.

Images