Providing a machine learning model based on a desired metric value

Abstract

The present disclosure relates to a method for generating a machine learning (ML) model. The method includes receiving a selection of an expected value of the metric for predicting a value of the first label attribute based on a current training dataset. A previously obtained set of training settings can be used to determine a set of training settings corresponding to the expected value of the metric. A ML engine can be controlled to generate a machine learning model using the current training dataset according to the determined set of training settings. The generated machine learning model can be deployed for performing the prediction of the value of the first label attribute.

Description

Background Technology

[0001] This invention relates to the field of digital computer systems, and more particularly, to a method for providing machine learning models based on desired metrics.

[0002] Machine learning models are integrated into many software systems, such as database transaction processing systems. The implementation of these models can be very complex. However, training and inference of such models may require tuning several parameters. Summary of the Invention

[0003] Various embodiments provide methods, computer systems, and computer program products as described in the independent claims. Embodiments of the invention can be freely combined with each other. The embodiments presented herein are not exhaustive. That is, many other embodiments, not mentioned, may exist within the scope and spirit of this disclosure.

[0004] Embodiments of the present invention may include computer-implemented methods, computer program products, and computer systems. Embodiments may allow defining metrics for evaluating the performance of machine learning (ML) models. Embodiments may also include providing an ML engine or similar computer program configured to receive a set of training settings and a training dataset including a first feature attribute and a first label attribute. Embodiments may further include generating at least one machine learning model based on the set of training settings and the training dataset. Embodiments may further include configuring the machine learning model to predict a first label attribute based on the first feature attribute. Further, embodiments may include providing values ​​for the generated machine learning model. Additionally, embodiments may include obtaining a set of training settings and associated values ​​of metrics from previous operations of the ML engine, and receiving a selection of an expected value for the metric for predicting the value of the first label attribute based on the current training dataset. Additionally, embodiments may include determining a set of training settings corresponding to the expected value of the metric based on the obtained set of training settings. Further, embodiments may include generating a machine learning model based on the determined set of training settings and the current training dataset. Finally, embodiments may include deploying the generated machine learning model to perform the prediction of the value of the first label attribute. Attached Figure Description

[0005] The embodiments of the invention are explained in more detail below with reference to the accompanying drawings, in which:

[0006] Figure 1 An exemplary computer system, generally designated 100, is shown according to an embodiment of the present disclosure.

[0007] Figure 2 An example set of settings is shown according to embodiments of this disclosure.

[0008] Figure 3 This is a flowchart of a method for providing a machine learning model according to embodiments of the present disclosure.

[0009] Figure 4 This is a flowchart illustrating a set of methods for determining the settings of a machine learning model, based on examples from this topic.

[0010] Figure 5A This is a flowchart of a method for determining a set of settings for a machine learning model according to embodiments of the present disclosure.

[0011] Figure 5B The graphical user interface output for training a machine learning model is shown according to an embodiment of the present disclosure.

[0012] Figure 5C A graphical user interface output for training a machine learning model according to an embodiment of the present disclosure is depicted.

[0013] Figure 6 An exemplary computing system, generally designated 600, capable of implementing one or more method steps according to an embodiment of the present disclosure, is shown.

[0014] Figure 7 A cloud computing environment according to an embodiment of the present invention is described.

[0015] Figure 8 An abstract model layer according to an embodiment of the present invention is described. Detailed Implementation

[0016] The description of different embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0017] Automated machine learning can be the process of automating the application of machine learning to real-world prediction problems using an ML engine. An ML engine can cover the entire pipeline from raw datasets to deployable machine learning models. An ML engine can be configured to generate one or more machine learning models from a training dataset based on a defined prediction problem in each experiment run. For this purpose, the ML engine can use a set of settings, which are configuration parameters such as a list of learning algorithms. These settings can be parameters and their associated values. Tuning settings can include changing the values ​​of the parameters in the settings. Each experiment run can be associated with experiment metadata describing that experiment run. The experiment metadata can, for example, describe the prediction problem. For this purpose, the experiment metadata can include attributes describing the prediction problem, such as attributes describing the number / type of feature attributes and attributes describing the label attributes of the training dataset. The ML engine can generate a model for predicting label attributes from one or more feature attributes. For example, a feature attribute could be an attribute describing a product, and a label attribute could indicate system satisfaction with the product. The ML engine can generate a model to predict satisfaction based on product attributes. In another example, the prediction problem could be predicting whether a new client system will subscribe to (yes / no) a term deposit based on demographic attributes associated with a client system.

[0018] While high automation in an ML engine allows for the use of machine learning models, appropriate settings for the ML engine are still required to achieve specific target performance. It may also be necessary to prevent the ML engine settings from being tuned multiple times to achieve the desired performance target. However, this can be challenging because the number of these settings can be very high (e.g., 100), making the task of discovering, training, and subsequently inferring the model tedious, lengthy, and computationally intensive. Furthermore, performance variations may be attributable solely to a few settings that might require further resource allocation. This topic addresses these issues by using a previous set of settings and the performance they achieve. This is advantageous because the previous set of settings can be obtained from several sources from which accurate results are obtained. According to this topic, the set of settings that the system can use can be a pre-tuned set of settings or settings that can be predicted by a predictive model. In both cases, the system may have to define a target performance value to obtain the desired machine learning model without tuning the settings. This saves processing resources that would otherwise be required several additional iterations to define and tune individual settings.

[0019] An ML engine can be used by multiple systems for generating machine learning models. This topic covers saving the processing inputs and outputs of an ML engine. For each prediction problem, a pair of metric values ​​and a corresponding set of training settings can be saved. For example, the first prediction problem could be predicting feature L1 from features F1 and F2, the second prediction problem could be predicting feature L2 from features F3, F4, and F5, and so on. After saving these pairs, the system can define one of the desired metric values ​​and prediction problems to obtain a set of training settings suitable for the system (e.g., the system can be configured to predict L1 from a new training dataset reflecting what the system is processing based on F1 and F2). In this case, the saved set of settings associated with the same prediction problem can be used to determine the set of settings used to generate a model from the new training dataset.

[0020] In an embodiment, determining the training settings set may include: creating a training set from entries in each training settings set. Each entry may include an input vector and an associated output vector. The input vector may include the value of a metric and metadata describing the attribute. The output vector may include the set of training settings. Additionally, the training sets are used to train a predictive model to predict the settings set by providing the expected value of the metric and metadata of the attributes of the current training dataset as input and receiving the determined settings set as output. The predictive model may be a multi-objective regression model.

[0021] A predictive model can be used to predict a set of settings for each desired metric. The system can provide the desired metric and a definition of the prediction question as input to the predictive model. The prediction question can be defined using experimental metadata describing feature and label attributes (e.g., their quantity and type). Following the example above, the system can be configured to predict a product's satisfaction level with 90% or better accuracy. In this case, the system can input the 90% expected accuracy value and the experimental metadata into the predictive model. The experimental metadata provides descriptions of feature attributes as product attributes and descriptions of label attributes as satisfaction. The predictive model can provide a set of settings as output, which the system can use as input to an ML engine to generate a machine learning model. This embodiment can be advantageous because it can provide accurate estimates of the settings. This can be particularly advantageous because the training set can be obtained from different sources with different attribute contents.

[0022] According to one embodiment, determining the set of settings may include selecting a set of settings from the obtained set of settings that correspond to the desired values ​​of a metric. This enables a simplified method for obtaining a set of settings from existing settings.

[0023] According to one embodiment, the training settings set may be a first subset of settings for preprocessing the training dataset, a second subset of settings for selecting one or more learning algorithms suitable for prediction, and a third subset of settings for training optimization.

[0024] A first subset of settings can be advantageous for the following reasons: the training dataset may have different data formats, duplicate values, and missing values, but the machine learning algorithm may need to work in a consistent or uniform format without missing values. To achieve this, the ML engine can apply different algorithms or estimators that analyze, clean, and prepare the training dataset. The first subset of settings controls this task of preprocessing the training dataset. An example setting for this first subset of settings could be a parameter called "drop_replicate," whose value can be set to "true" or "false," such that if it is "true," the preprocessing can involve deduplication of the training dataset.

[0025] A second subset can be advantageous for the following reasons: An ML engine can include a set of machine learning algorithms. Given a training dataset and a prediction problem, the ML engine can select one or more learning algorithms from this set to generate a model based on the given training dataset. For example, the ML engine can test and rank candidate algorithms against a small subset of the data, and progressively increase the size of the subset for the most promising algorithms to obtain the selected algorithms.

[0026] This third subset of settings can be advantageous for the following reasons. The number of hyperparameters in machine learning algorithms grows exponentially, making optimization beyond the capabilities of data science teams to perform manually and in a timely manner. ML engines can automate hyperparameter optimization, freeing teams from the burden of exploring and optimizing across the entire event space of hyperparameters. An example of a third subset of settings could be a flag indicating whether hyperparameter optimization is performed. Hyperparameters are parameters whose values ​​are used to control the learning process of a machine learning model (conversely, the values ​​of other parameters, such as node weights, are derived through training). Furthermore, the third subset of settings enables feature engineering. Feature engineering refers to a transformation process that derives new information from existing feature attributes; for example, feature attributes can be transformed into a combination of features that best represent the problem to achieve the most accurate predictions.

[0027] In an embodiment, the metadata describing the attributes (also known as experimental metadata) may include the number of feature attributes, the type of feature attributes, and / or the type of label attributes.

[0028] In embodiments, the method may further include standardizing the metric values ​​to a discrete range of values ​​with a predefined granularity. The obtained set of settings can be provided according to the granularity, such that each set in the obtained set of settings represents a corresponding granularity level. This range can be defined by a minimum value, a maximum value, and a step value or granularity. Providing a defined range enables uniform access to the set of settings. This can be particularly advantageous when multiple users are involved (e.g., providing users with a defined set of values ​​to choose from). For example, the range could be 0.0 to 1.0 with a step / granularity of 0.1. That is, the possible values ​​for the metric could be one of 10 values: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0. The values ​​of the metric can be standardized such that they fit within this range. Each of the 10 metric values ​​can be associated with one or more sets of training settings previously used by the ML engine. In this way, the system can select only the granularity level (e.g., 0.3) to obtain the corresponding set of training settings.

[0029] If a metric is a combination of more than one metric type (e.g., precision and runtime), the same range can be used to represent both metric types. For example, one metric type can have values ​​V within that range, while another metric type has values ​​1-V, and these metrics have been configured to sum to one (e.g., if the precision is 0.9, the runtime will be 0.1). This allows for efficient selection of metric values ​​in a multidimensional space, without having to select metrics individually.

[0030] In an embodiment, the receiving selection may include selection at the receiving granularity level.

[0031] In one embodiment, receiving a selection may include displaying a slider on a graphical user interface, allowing the user to set a value by moving the slider's indicator. The slider includes a range of standardized values, where the set value of the slider is the selected value of the received metric. The slider can effectively control the operation of the ML engine because the user of the system may only need to slide the indicator.

[0032] In embodiments, metrics may describe the runtime of training and / or inference of a machine learning model, including the accuracy and / or precision of the machine learning model, or a combination thereof. Precision gives a measure of the number of correct predictions made by the model. It can be the ratio of correct predictions to total predictions. Precision may indicate how correctly the model has detected positive results. It can be the ratio of true positives to total positives.

[0033] In an embodiment, a metric can be a combination of multiple metric types. Each metric type can be associated with a different set of settings from a set of settings. For example, a metric type can be precision, accuracy, runtime, etc. Runtime can be the time required to train the generated model. A combination of two metric types allows for the measurement of two metric types. When a metric includes a combination of two or more metric types, the selection of the metric value can be a value of at least one metric type and a saved / recorded set of settings associated with the value of the metric type. For example, if the metric is a combination of precision and runtime, the user can select a desired runtime value and / or a desired precision (e.g., each set of training settings can be associated with a pair of values ​​for precision and runtime).

[0034] In embodiments, the method may further include storing the determined set of settings and the values ​​of the metrics associated with them. For example, the determined set of settings may be stored in association with desired metric values ​​and experimental metadata, so that they can be used for future determination of metric values. This enables a self-improving system (i.e., a machine learning feedback loop).

[0035] Now refer to the attached diagram, Figure 1 A computer system 100 according to an embodiment of the present invention is shown. The computer system 100 includes a training system 101 and client systems 102a, 102b, and 102n. It should be noted that although in Figure 1 Three client systems are described, but any number of client systems (e.g., 1, 2, n…n+1) may exist in computer system 100. Training system 101 may be configured to communicate with each of client systems 102a-n via network 103. It should be noted that, for simplicity, only one network is shown; however, training system 101 may be connected to client systems 102a-n via more than one network (e.g., training system 101 may be connected to each of client systems 102a-n via a separate, respective network).

[0036] Network 103 may include, but is not limited to, cable networks, fiber optic networks, hybrid fiber-coaxial networks, wireless networks (e.g., Wi-Fi and / or mobile phone networks), satellite networks, the Internet, intranets, local area networks, and / or any combination of these networks.

[0037] As shown in the figure, the users 105a-n of the client systems 102a-n may include individuals, such as viewers, owners, technical support personnel, or devices, machine learning systems, or artificial intelligence systems. The training system 101 may be provided as a single server device or multiple server devices operating in a cloud computing configuration.

[0038] Training system 101 can be located remotely from client systems 102a-n and is accessible via network 103. Training system 101 can provide resources for training, operating, and storing machine learning models. The performance of the machine learning model can be quantified by defined metrics. A metric can be a single metric type or a combination of multiple metric types (e.g., a metric can be the runtime and / or precision and / or accuracy of the machine learning model). Training system 101 includes an ML engine 107. The ML engine can be configured to generate machine learning models (e.g., ...). The ML engine 107 is a computer program (AutoAI). For example, it can generate machine learning models for different prediction problems. Furthermore, the ML engine 107 can process the prediction problem for each experimental run. Additionally, the ML engine 107 can receive a set of training settings…and a training dataset including one or more feature attributes and label attributes in each experimental run. In some cases, experimental metadata can be provided. Experimental metadata describes the experimental run (e.g., experimental metadata describes the prediction problem).

[0039] Figure 2 An example of experimental metadata 210 is described. Experimental metadata 201 includes attributes describing the prediction problem and training data. Experimental metadata 201 includes attributes describing the prediction problem and training data. ML engine 107 can use the set of settings s1, s2…s m (like Figure 1 The training system 101 can generate one or more machine learning models (as shown) that can predict labels based on one or more features. Furthermore, each of the generated machine learning models can be associated with a value of a metric indicating the performance of the machine learning model. The training system 101 can be configured to record or save the activities of the ML engine 107. The training system 101 may further include a storage system 112. The storage system 112 can include PP1…PP1 for each prediction problem (PP) in the previously processed prediction problems. N Where N≥1, and tuples G, where each tuple can be a metric for one or more sets of setup and experiment metadata. For simplicity, Figure 1 This describes a set of settings for each metric. In the current case, the number of tuples, G, is the number of experimental runs. G can also represent the granularity of the metric values. Figure 1 As shown, the prediction problem PP1 includes tuples The tuples are related, the tuples Instructions and m1 settings set (in The upper index in the text refers to the prediction problem and experimental metadata. (exist The upper index in the context refers to the prediction problem and... The lower index in the table refers to the granularity level of the metric, associated with the metric value target1. The set of settings associated with the same prediction problem can have the same type and number of settings but different values ​​(e.g., ...). Figure 2 The set of settings shown can be the settings for prediction problem PP1. However, the values ​​assigned to each setting can be different for different experimental runs. The set of settings associated with a prediction problem can differ in number and / or type compared to another prediction problem. For example, the set of settings for prediction problem PP2 can be a subset of the set of settings for prediction problem PP1, and therefore they differ depending on the number of settings. The G-experimental runs associated with a prediction problem can involve the same attributes in the training dataset, but the contents of the training dataset (i.e., attribute values) can differ. For example, if the prediction problem is to predict feature L1 from features F1 and F2, all training datasets used in G-experimental runs for the same prediction problem can include at least three columns representing F1, F2, and L1. The contents of the three columns in the training dataset can differ.

[0040] For example, Figure 2 The set of training settings 200 shown can allow for a model with an accuracy of 0.89 and a runtime of 1028 seconds. The set of settings 200 includes a first subset 201 for preprocessing the training dataset, a second subset 202 for selecting one or more learning algorithms suitable for prediction, and a third subset 203 for training optimization. Furthermore, as in... Figure 2 As depicted, the first subset 202 includes the sample size of the training data set with a value of 0.5, and can indicate whether as described in... Figure 2 The settings described herein are used to deduplicate the training data.

[0041] Figure 3 This is a flowchart illustrating a method for generating machine learning models, based on examples from this topic. For illustrative purposes, in Figure 3 The method described in [the document] can be used in [the following context] Figure 1 The system shown is implemented, but is not limited to, this implementation. For example, Figure 3 The method can be executed by one of the client systems 102a-n or by the training system 101.

[0042] In step 301, the selection of the expected value for the metric can be received. Client system 102a can automatically select the expected value. For example, client system 102a can be configured to generate a model with an accuracy of 0.9 for a prediction problem previously processed by ML engine 107, namely the prediction problem PP1…PP. NThis is part of the process. An experimental run can be defined. In one exemplary embodiment of step 301, a graphical user interface (GUI) can be displayed to user 105a on client system 102a. User 105a of client system 102a can use the controls of the GUI to select a desired metric. The selected value can be sent to training system 101. User 105a can be a device, a machine learning system, or an artificial intelligence system.

[0043] Experimental metadata describing the experiment execution can be provided. Optionally, further instructions on the experimental metadata can be provided. The experimental metadata can describe the prediction problem for user 105a. The prediction problem can be the prediction problem PP1…PP previously processed by ML engine 107. N One of them. If no experimental metadata is indicated in step 301, the prediction problem PP1…PP may have already been classified as the default prediction problem. N One of these could be a prediction problem associated with the selection received. In both cases, for example, suppose that the current prediction problem being used by client system 102a is a previously processed prediction problem PP. N .

[0044] In step 303, the ML engine 107 determines a set of training settings corresponding to the expected values ​​of the metric based on the obtained set of training settings. For example, such as Figure 1 As shown, with the prediction problem PP N Associated G-tuples This can be used to determine the set of training settings corresponding to the desired metric selection values. In one example implementation of step 303, the desired metric value can be compared with the stored metric values ​​target1…target. G The set of settings associated with the metrics that match the selected metric can be the set of settings determined in step 303. Matching metric target1…ortarget G It can be equal to the selected metric or the closest metric to the selected metric. Figure 4 and 5A Other example implementations of step 303 are provided.

[0045] In step 305, ML engine 107 can generate a machine learning model based on one or more machine learning models trained on the current training dataset, according to the determined set of training settings.

[0046] For example, ML Engine 107 can receive a defined set of settings and the current training dataset, and can generate PPs for prediction problems. N One or more machine learning models.

[0047] The generated machine learning models (multiple models) can be deployed in step 307. This enables the prediction of the PP problem. N Perform predictions. For example, the generated machine learning model(s) can be provided to the client system 102a, allowing the system to infer the model.

[0048] Figure 4 It is used to determine the prediction problem for a given prediction problem (e.g., Figure 1 The flowchart of the method for generating the set of machine learning models (PP1). Figure 4 The method described in the document provides Figure 3 The implementation details of step 303.

[0049] In step 401, the training system 101 may create a training set. The training set may include an entry for each experimental run, used to obtain results for the prediction problem PP1, PP2…PP. N The training set consists of tuples. Each entry includes an input vector and an associated output vector. The input vector includes values ​​of the metrics used in the experimental run and experimental metadata describing the prediction problem (e.g., ...). The output vector includes a set of settings for the experimental run (e.g., ...). According to Figure 1 The example described herein indicates that the training set may include N×G entries:

[0050] In step 403, a prediction model can be trained using a training set for the prediction setting set. The prediction model can be provided in step 405, for example, it can be used to estimate... Figure 3 The set is set in step 303 of the method.

[0051] Figure 5A This is a flowchart illustrating a method for determining the set of settings for generating a machine learning model for a given prediction problem (e.g., PP1), based on examples from this topic. For illustrative purposes, in Figure 5A The method described in [the document] can be used in [the following context] Figure 1 The system shown is implemented, but is not limited to, this implementation. For example, Figure 5A The method can be executed by one of the client systems 102a-n.

[0052] The following can be displayed in step 501 of the client system 102a: Figure 5BThe graphical user interface 510 is shown. The graphical user interface 510 includes a slider 513. User 105a can use the slider 513 to select the desired values ​​for the runtime (speed) and accuracy of a machine learning model. The slider is split such that if, for example, the slider is set to a value of 0.9, it indicates an accuracy of 0.9 and a speed of 0.1; that is, the runtime and accuracy values ​​add up to 1. This allows for the selection of two metrics using a single slider.

[0053] When selecting the desired value of a metric in step 503, the client system 102a can provide and display, in step 505, a set of settings 515 corresponding to the desired value of the metric on the graphical user interface 510, such as... Figure 5C As shown. Setting set 515 can be, for example... Figure 2 The set of settings shown is 200.

[0054] Figure 6 This refers to a general computerized system 600 suitable for implementing at least a portion of the method steps involved in this disclosure.

[0055] It should be understood that the methods described herein are at least partially non-interactive and automated by computerized systems such as servers or embedded systems. However, in exemplary embodiments, the methods described herein can be implemented in (partially) interactive systems. These methods can be further implemented in software 612, firmware 622, hardware (processor) 605, or combinations thereof. In exemplary embodiments, the methods described herein are implemented as executable programs in software and executed by a dedicated or general-purpose digital computer, such as a personal computer, workstation, minicomputer, or mainframe computer. Thus, the most general system 600 includes a general-purpose computer 601.

[0056] In an exemplary embodiment, regarding the hardware architecture, such as Figure 6 As shown, computer 601 includes processor 605, memory (main memory) 610 coupled to memory controller 615, and one or more input and / or output (I / O) devices (or peripherals) 10, 645 communicatively coupled via local input / output controller 635. Input / output controller 635 may be, but is not limited to, one or more buses or other wired or wireless connections as known in the art. Input / output controller 635 may have additional elements omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communication. Furthermore, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components. As described herein, I / O devices 10, 645 may generally include any general-purpose cryptographic card or smart card known in the art.

[0057] Processor 605 is a hardware device for executing software (specifically, software stored in memory 610). Processor 605 may be any custom or commercially available processor, central processing unit (CPU), auxiliary processor among several processors associated with computer 601, semiconductor-based microprocessor (in the form of a microchip or chipset), or any device generally used for executing software instructions.

[0058] Memory 610 may include any or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM)), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM). Note that memory 610 may have a distributed architecture, in which different components are located far apart from each other but are accessible by processor 605.

[0059] The software 612 in memory 610 may include one or more separate programs, each program including an ordered list of executable instructions for implementing logical functions (particularly those involved in embodiments of the invention). Figure 6 In one example, the software 612 in memory 610 includes program instructions (e.g., instructions for managing a database such as a database management system).

[0060] The software in memory 610 should also typically include a suitable operating system (OS) 611. OS 611 essentially controls the execution of other computer programs, such as software 612 used to implement the methods described herein.

[0061] The method described herein can be in the form of a source program, an executable program (object code), a script, or any other entity including a set of instructions to be executed. When it is a source program, it needs to be translated by a compiler, assembler, interpreter, etc., which may or may not be included in memory 610 to operate appropriately in conjunction with OS 611. Furthermore, the method can be written in an object-oriented programming language with data and method classes, or in a procedural programming language with routines, subroutines, and / or functions.

[0062] In an exemplary embodiment, a conventional keyboard 650 and mouse 655 may be coupled to an input / output controller 635. Other output devices (such as I / O device 645) may include input devices, such as, but not limited to, printers, scanners, microphones, etc. Finally, I / O devices 10, 645 may further include devices for transmitting both input and output, such as, but not limited to, network interface cards (NICs) or modulators / demodulators (for accessing other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc. I / O devices 10, 645 may be any general-purpose cryptographic card or smart card known in the art. System 600 may further include a display controller 625 coupled to a display 630. In an exemplary embodiment, system 600 may further include a network interface for coupling to a network 665. Network 665 may be an IP-based network for communication between computer 601 and any external servers, clients, etc., via a broadband connection. Network 665 transmits and receives data between computer 601 and external system 30, which can be used to perform some or all of the steps of the methods discussed herein. In an exemplary embodiment, network 665 may be a managed IP network managed by a service provider. Network 665 may be implemented wirelessly, for example using wireless protocols and technologies such as WiFi, WiMax, etc. Network 665 may also be a packet-switched network, such as a local area network, wide area network, metropolitan area network, Internet network, or other similar type of network environment. Network 665 may be a fixed wireless network, wireless local area network (LAN), wireless wide area network (WWAN), personal area network (PAN), virtual private network (VPN), intranet, or other suitable network system, and includes devices for receiving and transmitting signals.

[0063] If the computer 601 is a PC, workstation, intelligent device, etc., the software in the memory 610 may also include a Basic Input / Output System (BIOS) 622. The BIOS is a collection of basic software routines that initialize and test the hardware at startup, boot the OS 611, and support data transfer between hardware devices. The BIOS is stored in ROM so that it can be executed when the computer 601 is activated.

[0064] When computer 601 is running, processor 605 is configured to execute software 612 stored in memory 610, transfer data to and from memory 610, and typically control the operation of computer 601 as described herein. The methods and OS 611 described herein (in whole or in part, but typically the latter) are read by processor 605, may be buffered within processor 605, and then executed.

[0065] When the systems and methods described herein are implemented in software 612, such as Figure 6 As shown, the method can be stored on any computer-readable medium (e.g., memory 620) for use by or in conjunction with any computer-related system or method. Memory 620 may include disk storage, such as HDD storage.

[0066] It should be understood that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0067] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0068] The features are as follows:

[0069] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0070] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0071] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0072] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0073] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0074] The service model is as follows:

[0075] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0076] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0077] Infrastructure as a Service (IaaS): The capabilities offered to consumers are processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0078] The deployment model is as follows:

[0079] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0080] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0081] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0082] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0083] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0084] See now Figure 7 The diagram illustrates an illustrative cloud computing environment 1050. As shown, the cloud computing environment 1050 includes one or more cloud computing nodes 1010 that can communicate with local computing devices used by cloud consumers, such as, for example, personal digital assistants (PDAs) or cellular phones 1054A, desktop computers 1054B, laptop computers 1054C, and / or automotive computer systems 1054N. The nodes 1010 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 1050 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 7 The types of computing devices 1054A-N shown are intended to be illustrative only, and computing node 1010 and cloud computing environment 1050 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0085] See now Figure 8 This demonstrates the 1050 cloud computing environment ( Figure 7 This provides a set of functional abstractions. It should be understood beforehand. Figure 8 The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:

[0086] The hardware and software layer 1060 includes hardware and software components. Examples of hardware components include: a mainframe 1061; a server 1062 based on a RISC (Reduced Instruction Set Computer) architecture; a server 1063; a blade server 1064; a storage device 1065; and network and networking components 1066. In some embodiments, software components include network application server software 1067 and database software 1068.

[0087] The virtualization layer 1070 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 1071; virtual storage 1072; virtual network 1073, including virtual private network; virtual application and operating system 1074; and virtual client 1075.

[0088] In one example, management layer 1080 may provide the following functionalities: Resource Provisioning 1081 Provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 1082 Provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security Provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 1083 Provides consumers and system administrators with access to the cloud computing environment. Service Level Management 1084 Provides cloud computing resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 1085 Provides pre-scheduling and procurement of cloud computing resources, anticipating future requirements for those resources according to the SLA.

[0089] Workload layer 1090 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 1091; software development and lifecycle management 1092; virtual classroom education delivery 1093; data analytics and processing 1094; transaction processing 1095; and training machine learning models based on metrics-based learning (MLT) 1096 according to this topic, for example, as referenced Figure 3 , Figure 4 or Figure 5A As described.

[0090] This topic may include the following entries.

[0091] Item 1. A computer-implemented method, comprising:

[0092] Define metrics for evaluating the performance of machine learning (ML) models;

[0093] Provide an ML engine, which is configured as follows:

[0094] Receive a set of training settings and a training dataset including a first feature attribute and a first label attribute;

[0095] Based on the set of training settings, at least one machine learning model is generated using the training dataset, the machine learning model being configured to predict the first label attribute from the first feature attribute;

[0096] Provide the value of the metric for the generated machine learning model;

[0097] Obtain a set of training settings for previous operations of the ML engine and associated values ​​for the metric;

[0098] Receive a selection of the expected value of the metric used to predict the value of the first label attribute based on the current training dataset;

[0099] Use the obtained set of training settings to determine the set of training settings corresponding to the expected value of the metric;

[0100] The ML engine is controlled to generate a machine learning model using the current training dataset based on a determined set of training settings; the generated machine learning model is deployed to perform predictions on the values ​​of the first label attributes.

[0101] Item 2. According to the method described in Item 1, the set of training settings includes:

[0102] Create a training set comprising entries for each training setting in the obtained training setting set, wherein each entry comprises an input vector and an associated output vector, the input vector comprising the value of the metric and metadata describing the attribute, and the output vector comprising the training setting set;

[0103] The training set is used to train a prediction model to predict the set of settings by providing the expected value of the metric and the metadata of the attribute of the current training dataset as input to the prediction model and receiving the determined set of settings as output.

[0104] Item 3: According to the method in Item 1, the determination of the set includes: selecting the set of settings corresponding to the expected value of the metric from the obtained set of settings.

[0105] Item 4. According to any one of Items 1 to 3, the set of training settings includes: a first subset of settings for preprocessing the training dataset, a second subset of settings for selecting one or more learning algorithms suitable for prediction, and a third subset of settings for training optimization.

[0106] Item 5. According to the method described in Item 2 or 4, the metadata describing the attribute includes at least one of the following: the number of feature attributes, the type of feature attributes, and the type of tag attributes.

[0107] Item 6: The method according to any one of items 1 to 5 above further includes standardizing the metric to a range having a predetermined granularity, wherein the obtained set of settings is provided according to the granularity, such that each set in the obtained set of settings represents a corresponding granularity level.

[0108] Item 7: According to the method of Item 6, wherein the selection of receivers includes the selection of receiver granularity level.

[0109] Item 8: According to the method described in Item 6 or 7, receiving a selection includes: displaying a graphical user interface including a slider, wherein a user can set a value using an indicator that moves the slider, the slider including a range of standardized values, wherein the set value of the slider is a selected value of the received metric.

[0110] Item 9: The method of any one of items 1 to 8 above, which measures the running time of training and / or inference of a machine learning model, the accuracy of a machine learning model, or a combination thereof.

[0111] Item 10. According to any one of Items 1 to 9, a metric is a combination of multiple metric types, wherein each metric type is associated with a different set of settings in a set of settings.

[0112] Item 11: The method according to any one of the preceding items 1 to 10 further includes recording the determined set of settings and the values ​​of the metrics associated therewith.

[0113] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0114] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0115] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0116] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0117] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0118] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0119] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

Claims

1. A computer-implemented method for training a machine learning model, comprising: Receive a set of training settings and a training dataset including a first feature attribute and a first label attribute; At least one machine learning model is generated based on the training dataset and the set of training settings, wherein the at least one machine learning model is configured to predict the first label attribute from the first feature attribute; Provide a value for a corresponding performance metric for the at least one generated machine learning model, wherein the corresponding performance metric corresponds to a corresponding runtime of the at least one generated machine learning model; Obtain the associated values ​​of one or more previously operated training settings sets and the corresponding performance metrics; The provided performance metric values ​​are standardized to a range with a predefined granularity, wherein the obtained set of settings is provided according to the granularity, and each set in the obtained set of training settings represents a corresponding granularity level; Displaying a graphical user interface, wherein the graphical user interface includes an input element that allows a user to select a step size value for the corresponding granularity level; The expected value for the corresponding performance metric is received by the user through interaction with the input element of the graphical user interface by receiving a selection of one of the corresponding granularity levels; Based on the obtained set of training settings and the associated values ​​of the corresponding performance metrics, a set of training settings corresponding to the expected values ​​of the corresponding performance metrics is determined. Generate a new machine learning model based on the determined set of training settings; and Deploy the generated new machine learning model.

2. The computer-implemented method according to claim 1, wherein, Determining the set of training settings includes: Create a training set, wherein the training set is at least one entry in each training setting set of the obtained training setting set, wherein each entry is an input vector and an associated output vector, wherein the input vector is the value of the metric and metadata describing the first feature attribute and the first label attribute, and the output vector is the training setting set; and A prediction model is trained based on the training set to predict the set of settings.

3. The computer-implemented method according to claim 1, wherein, Determining the set of settings includes: Receive a selection from the set of settings corresponding to the desired value of the metric in the obtained set of settings.

4. The computer-implemented method according to claim 1, wherein, The training settings set includes: a first subset of settings for preprocessing the training dataset, a second subset of settings for selecting one or more learning algorithms suitable for the prediction, and a third subset of settings for training optimization.

5. The computer-implemented method according to claim 2, wherein, The metadata describing the first feature attribute and the first tag attribute includes at least one of the following: the number of feature attributes, the type of the feature attributes, and the type of the tag attribute.

6. The computer-implemented method according to claim 1, wherein, The input element of the graphical user interface includes a slider for setting the desired value of the corresponding performance metric for the runtime.

7. The computer-implemented method according to claim 1, wherein, The metric is a combination of two or more metric types, wherein each metric type is associated with a different set of settings in the set of settings.

8. The computer-implemented method according to claim 1, further comprising: Record the determined set of settings and the value of the metric associated with the determined set of settings.

9. The computer-implemented method according to claim 1, further comprising: Create a training set comprising entries for each training setting in the obtained training settings set, wherein each entry comprises an input vector and an associated output vector, the input vector comprising the value of the metric and metadata describing the first feature attribute and the first label attribute, and the output vector comprising the training settings set; and The training set is used to train a prediction model to predict the set of settings by providing the expected value of the metric and metadata of the attribute of the current training dataset as input to the prediction model and receiving the determined set of settings as output.

10. A computer system, the computer system comprising: processor; Computer-readable storage medium; as well as Computer program instructions stored on the computer-readable storage medium, the computer program instructions being used for: Receive a set of training settings and a training dataset including a first feature attribute and a first label attribute; At least one machine learning model is generated based on the training dataset and the set of training settings, wherein the at least one machine learning model is configured to predict the first label attribute from the first feature attribute; Provide the value of a corresponding performance metric for the at least one generated machine learning model, wherein the corresponding performance metric corresponds to the corresponding runtime of the at least one generated machine learning model; Obtain the associated values ​​of one or more previously operated training settings sets and the corresponding performance metrics; The provided performance metric values ​​are standardized to a range with a predefined granularity, wherein the obtained set of settings is provided according to the granularity, and each set in the obtained set of training settings represents a corresponding granularity level; Displaying a graphical user interface, wherein the graphical user interface includes an input element that allows a user to select a step size value for the corresponding granularity level; The expected value for the corresponding performance metric is received by the user through interaction with the input element of the graphical user interface by receiving a selection of one of the corresponding granularity levels; Based on the obtained set of training settings and the associated values ​​of the corresponding performance metrics, a set of training settings corresponding to the expected values ​​of the corresponding performance metrics is determined. Generate a new machine learning model based on the determined set of training settings; and Deploy the generated new machine learning model.

11. The computer system according to claim 10, wherein, The training setup set is determined to include computer program instructions, which are used to: Create a training set, wherein the training set is at least one entry in each training setting set of the obtained training setting set, wherein each entry is an input vector and an associated output vector, wherein the input vector is the value of the metric and metadata describing the first feature attribute and the first label attribute, and the output vector is the training setting set; and A prediction model for predicting the set of settings is trained based on the training set.

12. The computer system according to claim 10, wherein, The set of settings is determined to include computer program instructions, which are used for: Receive a selection from the set of settings corresponding to the desired value of the metric in the obtained set of settings.

13. The computer system according to claim 10, wherein, The training settings set includes: a first subset of settings for preprocessing the training dataset, a second subset of settings for selecting one or more learning algorithms suitable for the prediction, and a third subset of settings for training optimization.

14. The computer system according to claim 11, wherein, The metadata describing the first feature attribute and the first tag attribute includes at least one of the following: multiple feature attributes, the type of the feature attributes, and the type of the tag attribute.

Images