Providing machine learning models based on desired metric values.

Abstract

The present disclosure relates to a method for generating a machine learning (ML) model, the method including receiving a selection of a desired value of a metric for predicting a value of a first label attribute based on a current training dataset. A set of previously obtained training settings can be used to determine a set of training settings corresponding to the desired value of the metric. An ML engine can be controlled to generate a machine learning model according to the determined set of training settings using the current training dataset. The generated machine learning model can be deployed to make predictions of the value of the first label attribute.

Description

[Technical Field]

[0001] The present invention relates to the field of digital computer systems, and more specifically, to a method for providing a machine learning model based on a desired metric value. [Background technology]

[0002] Machine learning models are increasingly being incorporated into many software systems, such as database transaction processing systems. These models can be extremely complex to implement. However, training and inference with such models may require tuning several parameters. [Overview of the Initiative]

[0003] Various embodiments provide methods, computer systems, and computer program products described by the subject matter of the independent claims. Embodiments of the present invention can be freely combined with one another. The embodiments presented herein are not exhaustive. Therefore, there may be numerous additional embodiments not described herein that are included in the scope and spirit of this disclosure.

[0004] Embodiments of the present invention may include computer implementation methods, computer program products, and computer systems. Embodiments may enable the definition of metrics for evaluating the performance of machine learning (ML) models. Embodiments may further include providing an ML engine or similar computer program configured to receive a set of training settings and a training dataset containing first feature attributes and first label attributes. Embodiments may further include generating at least one machine learning model according to the set of training settings and the training dataset. Embodiments may further include configuring the machine learning model to predict first label attributes from first feature attributes. Embodiments may also further include providing metric values ​​for the generated machine learning model. Furthermore, embodiments may include obtaining the set of training settings and relevant values ​​of the metric from a previous operation of the ML engine and receiving a selection of desired values ​​for the metric to predict the value of the first label attribute based on the current training dataset. Furthermore, embodiments may include determining a set of training settings corresponding to desired values ​​for the metric based on the obtained set of training settings. Embodiments may also include generating a machine learning model based on the determined set of training settings and the current training dataset. Finally, the embodiment may include deploying the generated machine learning model to predict the value of the first label attribute.

[0005] Hereinafter, embodiments of the present invention will be described in more detail with reference to the drawings, merely as examples. [Brief explanation of the drawing]

[0006] [Figure 1] This figure shows an exemplary computer system according to an embodiment of the present disclosure, with the whole system represented as 100. [Figure 2]This figure shows an example set of settings according to an embodiment of the present disclosure. [Figure 3] This is a flowchart showing a method for providing a machine learning model according to embodiments of the present disclosure. [Figure 4] This flowchart shows a method for determining a set of settings for generating a machine learning model, according to one embodiment of this subject. [Figure 5A] This flowchart shows a method for determining a set of settings for generating a machine learning model according to embodiments of the present disclosure. [Figure 5B] This figure shows a graphical user interface output for training a machine learning model according to an embodiment of the present disclosure. [Figure 5C] This figure shows a graphical user interface output for training a machine learning model according to an embodiment of the present disclosure. [Figure 6] This figure shows an exemplary computer system, shown in its entirety at 600, which can implement one or more method steps according to embodiments of the present disclosure. [Figure 7] This figure shows a cloud computing environment according to one embodiment of the present invention. [Figure 8] This figure shows an abstraction model layer according to one embodiment of the present invention. [Modes for carrying out the invention]

[0007] Various embodiments of the present invention are described for illustrative purposes, but are not intended to be exhaustive or limit to the embodiments disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments described. The terminology used herein has been selected to best describe the principles of the embodiments, their practical applications, or the technical improvements to the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0008] Automated machine learning can be the process of automating the task of applying machine learning to real-world prediction problems using an ML engine. An ML engine can cover the entire pipeline from raw datasets to obtaining deployable machine learning models. An ML engine may be configured to generate one or more machine learning models from a training dataset based on a defined prediction problem in each experimental run. To this end, an ML engine may use a set of configuration parameters, which may be a list of learning algorithms. These configurations may be parameters and their associated values. Tuning a configuration may involve changing the values ​​of the configuration parameters. Each experimental run may be associated with experimental metadata that describes that run. The experimental metadata may, for example, describe the prediction problem. To this end, the experimental metadata may include attributes that describe the prediction problem, such as attributes describing the number / type of feature attributes and attributes describing the label attributes of the training dataset. An ML engine can generate a model to predict label attributes from one or more feature attributes. For example, feature attributes may be attributes that describe a product, and label attributes may represent system satisfaction with the product. An ML engine can generate a model to predict satisfaction based on product attributes. In another embodiment, the prediction problem can be used to predict whether a new client system will apply for a time deposit (yes / no) based on demographic attributes associated with the client system.

[0009] While advanced automation in ML engines can enable the use of machine learning models, there is still a need to provide the appropriate configuration of the ML engine to achieve a specific target performance. This may require additional adjustments to avoid having to fine-tune the ML engine configuration multiple times to achieve the desired performance target. However, since the number of these configurations can be very large (e.g., 100), this can be difficult, making the task of finding, training, and then inferring a tedious, time-consuming, and computationally intensive task. Furthermore, performance can vary significantly with just a few configurations, necessitating the identification of additional resources. This subject can address these challenges by using a set of previous configurations and their achieved performance. This can be advantageous because the set of previous configurations may be derived from several sources from which accurate results were obtained. The set of configurations that the system can use according to this subject can be a pre-tuned set or can be predicted by a predictive model. In both cases, the system may need to define target performance values ​​to obtain the desired machine learning model without tweaking the configurations. This saves processing resources that would otherwise be required by several iterations to define and tune individual configurations.

[0010] ML engines can be used by numerous systems to generate machine learning models. This subject makes it possible to save the processing inputs and outputs of an ML engine. For each prediction problem, a pair of metric values ​​and a corresponding set of training settings can be saved. For example, the first prediction problem might be to predict feature L1 from features F1 and F2, and the second prediction problem type might be to predict feature L2 from features F3, F4, and F5. After saving these pairs, the system can define a desired set of metric values ​​and one of the prediction problems to obtain a set of training settings suitable for that system (for example, the system can be configured to predict L1 from F1 and F2 from a new training dataset that reflects what the system is processing). In this case, the saved set of settings associated with the same prediction problem can be used to determine the set of settings for generating a model from a new training dataset.

[0011] In one embodiment, determining a set of training settings may include creating a training set from entries in each set of training settings. Each entry may include an input vector and an associated output vector. The input vector may include metric values ​​and metadata describing attributes. The output vector may include a set of training settings. Furthermore, the predictive model may be trained using the training set to predict the set of settings by providing desired metric values ​​and metadata for attributes of the current training dataset as input to the predictive model and receiving the determined set of settings as output. The predictive model may be a multi-target regression model.

[0012] A predictive model can be used to predict a set of settings for each desired metric value. The system can provide the desired metric value and a definition of the prediction problem as input to the predictive model. The prediction problem can be defined by experimental metadata that describes feature and label attributes (e.g., their number and type). Following the example above, the system can be configured to predict product satisfaction with an accuracy of 90% or higher. In this case, the system can input the desired accuracy value of 90% and experimental metadata into the predictive model. The experimental metadata provides a description of the feature attribute, which is an attribute of the product, and a description of the label attribute, which is satisfaction. The predictive model can provide as output a set of settings that the system can use as input to an ML engine to generate a machine learning model. This embodiment can be advantageous because it can provide an accurate estimation of the settings. This can be particularly advantageous because training sets can be obtained from different sources containing different content of attributes.

[0013] According to one embodiment, determining a set of settings may include selecting a set of settings from a set of acquired settings that corresponds to a desired value of the metric. This can enable a simplified method of obtaining a set of settings from existing settings.

[0014] According to one embodiment, the set of training settings may be a first subset of settings for preprocessing the training dataset, a second subset of settings for selecting one or more learning algorithms suitable for prediction, and a third subset of settings for training optimization.

[0015] The first subset of settings is advantageous for the following reasons. The training dataset may have different data formats, duplicate values, and missing values, but the machine learning algorithm may need to function with a consistent format or the same format and without missing values. To enable this, the ML engine can apply various algorithms or estimators that can analyze, clean, and prepare the training dataset. The first subset of settings can control this task of preprocessing the training dataset. An exemplary setting of the first subset of settings is a parameter named "drop_duplicate" that can set a value to "true" or "false", and as a result, if the value is "true", this preprocessing can include duplicate elimination of the training dataset.

[0016] The second subset of settings is advantageous for the following reasons. The ML engine can include a set of machine learning algorithms. For a given training dataset and prediction problem, the ML engine can select one or more learning algorithms from that set of algorithms to generate a model based on that given training dataset. For example, the ML engine can test and rank candidate algorithms against a small subset of the data and gradually increase the size of the subset for the most promising algorithms to arrive at the selected algorithm.

[0017] The third subset of settings can be advantageous for the following reasons. The number of hyperparameters of a machine learning algorithm grows exponentially, which exceeds the ability of a data science team to manually and timely achieve the optimization of the machine learning algorithm. The ML engine can enable automated hyperparameter optimization (HPO), which reduces the intensive responsibility of the team to search and optimize across the complete event space for hyperparameters. An example of the third subset of settings is a flag to indicate whether to perform hyperparameter optimization or not. Hyperparameters are parameters whose values are used to control the learning process of a machine learning model (in contrast, the values of other parameters such as node weights are derived through training). Additionally, the third subset of settings can enable feature engineering. Feature engineering relates to a transformation process for deriving new information from existing feature attributes. For example, feature attributes can be transformed into a combination of features that best represent the problem in order to achieve the most accurate prediction.

[0018] In one embodiment, metadata (also referred to as experimental metadata) describing the attributes can include the number of feature attributes, the type of feature attributes or label attributes, or a combination thereof.

[0019] In one embodiment, this method may further include normalizing the metric value to a range of discrete values ​​having a predefined granularity. The acquired set of settings can be provided according to the granularity, and as a result, each set of acquired settings represents a corresponding granularity level. This range can be defined by a minimum value, a maximum value, and a step value or granularity. Providing a defined range can enable uniform access to the set of settings, which can be particularly advantageous when many users are involved (for example, users are provided with a defined set of values ​​from which to choose). For example, this range could be 0.0 to 1.0 with a step / granularity of 0.1. That is, the possible values ​​of the metric may be one of 10 values: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0. The metric value can be normalized so that the metric value falls within that range. Each of these 10 metric values ​​can be associated with one or more sets of training settings previously used by the ML engine. In this way, the system can obtain a corresponding set of training settings simply by selecting a granularity level (e.g., 0.3).

[0020] If a metric is a combination of two or more metric types (e.g., precision and runtime), the same range can be used to represent two of the metric types. For example, one metric type may have a range value V, while another metric type has a value 1-V, and the metrics are configured to combine to 1 (for example, if precision is 0.9, then runtime will be 0.1). This can enable efficient selection of metric values ​​in a multidimensional space, eliminating the need to select metrics individually.

[0021] In one embodiment, receiving a selection may include receiving a selection at a granular level.

[0022] In one embodiment, receiving a selection may include displaying a slider on a graphical user interface, the slider allowing the user to set a value by moving the slider's indicator, the slider containing a normalized range of values, and the set value of the slider being the selected and received value of the metric. The slider can efficiently control the operation of the ML engine because the system user only needs to slide the indicator.

[0023] In one embodiment, the metrics can describe the training or inference runtime of a machine learning model, or both, including the accuracy or precision of the machine learning model, or both, or a combination thereof. Accuracy gives a measure of the number of accurate predictions made by the model. Accuracy can be the ratio of accurate predictions to total predictions. Precision can indicate how accurately the model detected positive outcomes. Precision can be the ratio of true positives to total positives.

[0024] In one embodiment, a metric can be a combination of multiple metric types. Each metric type can be associated with a separate set of settings from a set of settings. For example, metric types could be precision, accuracy, runtime, etc. Runtime can be the time required to train the generated model. A combination of two metric types allows for the measurement of both metric types. For a metric that includes a combination of two or more metric types, the selection of metric values ​​can be a value for at least one metric type and a set of saved / recorded settings associated with the value of the metric type. For example, if the metric is a combination of precision and runtime, the user can select a desired runtime value or a desired precision or both (for example, each set of training settings can be associated with a pair of precision and runtime values).

[0025] In one embodiment, this method may further include storing a set of determined settings and the associated metric values. For example, the set of determined settings can be stored in relation to desired metric values ​​and experimental metadata, and thus they can be used for future determination of metric values. This can enable a self-improving system (i.e., a machine learning feedback loop).

[0026] Next, looking at the figure, Figure 1 shows a computer system 100 according to one embodiment of the present invention. The computer system 100 includes a training system 101 and client systems 102a, 102b, and 102n. Although three client systems are shown in Figure 1, it should be noted that the computer system 100 may have any number of client systems (e.g., one, two, n...n+1). The training system 101 can be configured to communicate with each of the client systems 102a-n via a network 103. For simplicity of explanation, only one network is shown, but it should be noted that the training system 101 may connect to the client systems 102a-n via two or more networks (for example, the training system 101 may connect to each of the client systems 102a-n via separate networks).

[0027] Network 103 may include, but is not limited to, cable networks, fiber optic networks, hybrid fiber coaxial networks, wireless networks (e.g., WiFi or cellular networks or both), satellite networks, the Internet, intranets, local area networks, or combinations thereof.

[0028] As shown in the diagram, users 105a to n of client systems 102a to n may include individuals such as viewers, owners, and support technicians, or devices, machine learning systems, or artificial intelligence systems. The training system 101 may be provided as a single server device or multiple server devices operating in a cloud computing configuration.

[0029] The training system 101 may be remote to client systems 102a-102n and can be accessed via network 103. The training system 101 can provide resources for training, running, and storing machine learning models. The performance of the machine learning models can be quantified by defined metrics. The metrics can be one metric type or a combination of multiple metric types (for example, the metrics can be the runtime, accuracy, or precision of the machine learning model, or a combination thereof). The training system 101 includes an ML engine 107. The ML engine is a computer program that can be configured to generate machine learning models, such as IBM(R) AutoAI. For example, the ML engine 107 can generate machine learning models for different prediction problems. Furthermore, the ML engine 107 can process prediction problems per experimental run. Furthermore, for each experimental run, the ML engine 107 can receive a set of training settings s_1, s_2...s_m and a training dataset containing one or more feature attributes and label attributes. In some cases, experimental metadata can be provided. Experiment metadata describes the experimental run (for example, experimental metadata describes the prediction problem).

[0030] Figure 2 shows an example of experimental metadata 210. Experimental metadata 201 includes attributes describing the prediction problem and training data. The ML engine 107 can generate one or more machine learning models capable of predicting labels from one or more features using a set of settings s_1, s_2...s_m (shown in Figure 1). Furthermore, each of the generated machine learning models may be associated with a metric value indicating the performance of the machine learning model. The training system 101 may be configured to record or store the activity of the ML engine 107. The training system 101 may further include a storage system 112. The storage system 112 stores previously processed prediction problems PP1...PP where N≧1. N For each prediction problem (PP), there can be G tuples, each tuple can be a metric value and experimental metadata for one or more sets of settings. For simplicity, Figure 1 shows one set of settings for each metric value. In this case, the number of tuples G is the number of experimental runs. G can also represent the granularity of the metric values. As shown in Figure 1, prediction problem PP1 is associated with tuples, and those tuples are tuples

number

number

number

number

number

[0031] For example, the training configuration set 200 shown in Figure 2 can consider a model with an accuracy value of 0.89 and a runtime value of 1028 seconds. The configuration set 200 includes a first subset 201 of configurations for preprocessing the training dataset, a second subset 202 of configurations for selecting one or more learning algorithms suitable for prediction, and a third subset 203 of configurations for training optimization. Furthermore, as shown in Figure 2, the first subset 202 of configurations includes a sample size of training data set to a value of 0.5, which can indicate whether or not to deduce duplicate training data as shown in Figure 2.

[0032] Figure 3 is a flowchart illustrating a method for generating a machine learning model, as an example of this subject. For illustrative purposes, the method described in Figure 3 can be implemented in the system shown in Figure 1, but is not limited to this implementation. The method in Figure 3 can be performed, for example, by one of the client systems 102a to n or by the training system 101.

[0033] In step 301, the selection of the desired value for the metric can be received. The desired value can be automatically selected by the client system 102a. For example, the client system 102a can receive the prediction problem PP1...PP previously processed by the ML engine 107. N It can be configured to generate a model with an accuracy of 0.9 for the prediction problem, which is part of the process. An experimental run can be defined. In one exemplary implementation of step 301, a graphical user interface can be displayed to user 105a on client system 102a. User 105a of client system 102a can select a desired value for a metric using the control elements of the graphical user interface. The selected value can be sent to training system 101. User 105a can be a device, a machine learning system, or an artificial intelligence system.

[0034] Experiment metadata describing the experimental run can be provided. Optionally, this selection may further indicate this experimental metadata. This experimental metadata may describe the prediction problem of user 105a. This prediction problem is a prediction problem PP1...PP previously processed by the ML engine 107. N It can be one of the following. If experimental metadata is not shown in step 301, it can be classified as the default prediction problem PP1...PP NOne of them can be a prediction problem related to the received selection. In both cases, for example, the current prediction problem used by client system 102a is the previously processed prediction problem PP N is assumed to be.

[0035] In step 303, the ML engine 107 determines a set of training settings corresponding to the desired value of the metric based on the set of acquired training settings. For example, as shown in FIG. 1, G tuples N related to the prediction problem PP

Number

Number

[0036] In step 305, the ML engine 107 can generate a machine learning model according to the determined set of training settings based on one or more machine learning models trained on the current training dataset. The current training dataset can at least include columns representing the feature attributes and label attributes of the current prediction problem PP N .

[0037] For example, ML engine 107 can receive a set of determined settings and the current training dataset, and predict the problem PP. N It is possible to generate one or more machine learning models for [the given condition].

[0038] In step 307, the generated machine learning model can be deployed. This is for the prediction problem PP. N This makes it possible to make predictions according to the model. For example, the generated machine learning model can be provided to the client system 102a so that the system can infer the model.

[0039] Figure 4 is a flowchart showing how to determine a set of settings for generating a machine learning model for a given prediction problem (e.g., PP1 in Figure 1). The method shown in Figure 4 provides details of an implementation of step 303 in Figure 3.

[0040] In step 401, the training system 101 can create a training set. The training set consists of prediction problems PP1, PP2...PP N It can include per-experiment run entries from the experimental runs used to obtain a tuple for the training set. Each entry in the training set includes an input vector and an associated output vector. The input vector contains the metric values ​​for that experimental run and experimental metadata describing the prediction problem (for example).

number

number

number

number

[0041] In step 403, a predictive model can be trained using a training set to predict a set of settings. In step 405, this predictive model can be provided, and for example, it can be used to estimate the set of settings in step 303 of the method in Figure 3.

[0042] Figure 5A is a flowchart illustrating a method for determining a set of settings to generate a machine learning model for a given prediction problem, e.g., PP1, according to an example of this subject. For illustrative purposes, the method described in Figure 5A can be implemented in the system shown in Figure 1, but is not limited to this implementation. The method in Figure 5A can be performed, for example, by one of the client systems 102a to n.

[0043] In step 501, a graphical user interface 510, as shown in Figure 5B, can be displayed on the client system 102a. The graphical user interface 510 includes a slider 513. User 105a can use the slider 513 to select desired values ​​for the runtime (speed) and accuracy of the machine learning model. This slider is divided such that, for example, when the slider is set to a value of 0.9, this indicates an accuracy of 0.9 and a speed of 0.1, i.e., the runtime value and accuracy value add up to 1. This makes it possible to select two metric values ​​using a single slider.

[0044] After selecting the desired value for the metric in step 503, the client system 102a may, in step 505, provide and display a set of settings 515 corresponding to the desired value for the metric on the graphical user interface 510, as shown in Figure 5C. The set of settings 515 could be, for example, the set of settings 200 shown in Figure 2.

[0045] Figure 6 shows a general-purpose computerized system 600 suitable for implementing at least some of the method steps relating to this disclosure.

[0046] It should be understood that the methods described herein are at least partially non-interactive and are automated by a computerized system such as a server or an embedded system. However, in the exemplary embodiments, the methods described herein can be implemented in a (partially) interactive system. These methods can further be implemented by software 612, firmware 622, hardware (processor) 605, or a combination thereof. In the exemplary embodiments, the methods described herein are implemented in software as executable programs and run by a dedicated or general-purpose digital computer such as a personal computer, workstation, minicomputer, or mainframe computer. Thus, the most general-purpose system 600 includes a general-purpose computer 601.

[0047] In the exemplary embodiment, as shown in Figure 6, the hardware architecture of the computer 601 includes a processor 605, memory (main memory) 610 coupled to a memory controller 615, and one or more input / output (I / O) or both devices (or peripherals) 10, 645 communicably coupled via a local input / output controller 635. The input / output controller 635 may be one or more buses or other wired or wireless connections as known in the art, but is not limited to these. The input / output controller 635 may have additional elements omitted for simplicity of the figure, such as controllers, buffers (caches), drivers, repeaters, and receivers that enable communication. The local interface may also include address connections, control connections, or data connections, or a combination thereof, to enable proper communication between the above components. The I / O devices 10, 645 described herein may generally include any general-purpose cryptographic card or smart card known in the art.

[0048] The processor 605 is, in particular, a hardware device for executing software stored in memory 610. The processor 605 can be any custom-made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with computer 601, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally any device for executing software instructions.

[0049] Memory 610 may include one or a combination of volatile memory elements (e.g., random access memory (RAM such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM)), electrically erasable programmable read-only memory (EEPROM), and programmable read-only memory (PROM). Note that memory 610 may have a distributed architecture in which various components are remote to each other but can be accessed by processor 605.

[0050] The software 612 in memory 610 may contain one or more separate programs, each containing an ordered list of executable instructions for implementing logical functions, in particular functions relating to embodiments of the present invention. In the embodiment of Figure 6, the software 612 in memory 610 contains program instructions (for example, instructions for managing a database, such as a database management system).

[0051] The software in memory 610 typically also needs to include a suitable operating system (OS) 611. The OS 611 essentially controls the execution of other computer programs, such as software 612, to implement the methods described herein.

[0052] The methods described herein may take the form of a source program, an executable program (object code), a script, or any other entity including a set of instructions to be executed. In the case of a source program, the program needs to be translated by a compiler, assembler, interpreter, etc., which may or may not be contained in memory 610, so that it can function properly in conjunction with OS611. Alternatively, a method may be written as an object-oriented programming language having classes of data and methods, or as a procedural programming language having routines, subroutines, or functions, or a combination thereof.

[0053] In the exemplary embodiment, a conventional keyboard 650 and a mouse 655 can be coupled to the input / output controller 635. Other output devices, such as I / O device 645, may include input devices, such as printers, scanners, and microphones, but are not limited to these. Finally, I / O devices 10, 645 may further include devices that transmit both input and output, such as network interface cards (NICs) or modulators / demodulators (for accessing other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc. I / O devices 10, 645 may also be any general-purpose cryptographic card or smart card known in the art. System 600 may further include a display controller 625 coupled to a display 630. In the exemplary embodiment, system 600 may further include a network interface for coupling to a network 665. Network 665 may be an IP-based network for communication between computer 601 and any external server, client, etc., via a broadband connection. Network 655 transmits and receives data between computer 601 and external system 30 that may be necessary to perform some or all of the steps of the method described herein. In exemplary embodiments, network 665 may be a managed IP network managed by a service provider. Network 665 may be implemented wirelessly using wireless protocols and technologies such as WiFi, WiMAX, etc. Network 665 may also be a packet-switched network such as a local area network, wide area network, metropolitan area network, internet network, or other similar type of network environment.Network 665 may be a fixed wireless network, a wireless local area network (WLAN), a wireless wide area network (WWAN), a personal area network (PAN), a virtual private network (VPN), an intranet, or other suitable network system, and may include equipment for sending and receiving signals.

[0054] If the computer 601 is a PC, workstation, intelligent device, etc., the software in memory 610 may also include a Basic Input / Output System (BIOS) 622. The BIOS is a set of basic software routines that initialize and test the hardware at startup, start the OS 611, and assist in the transfer of data between hardware devices. The BIOS is stored in ROM so that it can be executed when the computer 601 is started.

[0055] When computer 601 is operating, processor 605 is configured to execute software 612 stored in memory 610, exchange data with memory 610, and generally control the operation of computer 601 according to the software. The methods described herein and OS 611 are read by processor 605, possibly buffered within processor 605, and then executed, in whole or in part, typically in part.

[0056] When the systems and methods described herein are implemented in software 612 as shown in Figure 6, the methods may be stored in any computer-readable medium, such as storage 620, for use by any computer-related system or method, or for use related thereto. Storage 620 may include disk storage such as HDD storage.

[0057] While this disclosure includes a detailed description of cloud computing environments, it should be understood that implementations of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented with any other type of computing environment that is currently known or may be developed in the future.

[0058] Cloud computing is a service distribution model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0059] The features are as follows:

[0060] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing functions such as server time and network storage as needed, without requiring human intervention from a service provider.

[0061] Wide network access: The functionality is available over the network and can be accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, and PDAs).

[0062] Resource Pooling: To accommodate multiple consumers using a multi-tenant model, a provider's computing resources are pooled, and different physical and virtual resources are dynamically allocated and reallocated as needed. Consumers generally have no control over or knowledge of the exact location of the resources provided, but may be able to specify a higher level of abstraction (e.g., country, state, or data center), which gives a sense of location independence.

[0063] Rapid scalability: With rapid scalability, features can be provisioned automatically in some cases, allowing for rapid scaling out, and features can be quickly released to scale in. To consumers, the features available for provisioning often appear endless, and they can purchase as many as they like, whenever they want.

[0064] Metered Services: Cloud systems automatically control and optimize resource utilization by leveraging appropriate levels of metric functionality depending on the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers regarding the services used.

[0065] The service model is as follows:

[0066] Software as a Service (SaaS): The functionality provided to consumers is the use of a provider's applications running on cloud infrastructure. These applications are accessible from various client devices via thin-client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or individual application functions, with the possible exception of limited user-specific application configuration settings.

[0067] Platform as a Service (PaaS): The functionality offered to consumers is the ability to deploy consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, or storage, but they can control the deployed applications and, in some cases, the application hosting environment configuration.

[0068] Infrastructure as a Service (IaaS): The functionality provided to consumers is the provisioning of processing, storage, networking, and other basic computing resources, allowing consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they can control the operating system, storage, and deployed applications, and in some cases, have limited control over selected network components (e.g., host firewalls).

[0069] The deployment model is as follows:

[0070] Private Cloud: This cloud infrastructure is operated solely for the organization. It can be managed by the organization or a third party and can reside on-premises or off-premises.

[0071] Community Cloud: This cloud infrastructure is shared by several organizations to support specific communities with common interests (e.g., missions, security requirements, policies, and compliance matters). It can be managed by an organization or a third party and can reside on-premises or off-premises.

[0072] Public Cloud: This cloud infrastructure is available to the public or large industry groups and is owned by the organization that sells the cloud services.

[0073] Hybrid Cloud: This cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain separate entities but are connected by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0074] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure, including a network of interconnected nodes.

[0075] Referring now to Figure 7, an illustrative cloud computing environment 1050 is illustrated. As shown in the figure, the cloud computing environment 1050 includes one or more cloud computing nodes 1010 that can communicate with local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 1054A, a desktop computer 1054B, a laptop computer 1054C, or an automotive computer system 1054N, or a combination thereof. The nodes 1010 can communicate with each other. The nodes 1010 may be physically or virtually grouped (not shown) in one or more networks, such as the private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof. This allows the cloud computing environment 1050 to provide infrastructure, platform, or software, or a combination thereof, as a service that does not require cloud consumers to maintain resources on their local computing devices for that purpose. The types of computing devices 1054A to 1054N shown in Figure 7 are for illustrative purposes only, and it is understood that the computing node 1010 and the cloud computing environment 1050 can communicate with any type of computerized device via any type of network connection or network addressable connection or a combination thereof (for example, using a web browser).

[0076] Referring now to Figure 8, a set of functional abstraction layers provided by the cloud computing environment 1050 (Figure 7) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 8 are for illustrative purposes only, and embodiments of the present invention are not limited thereto. As shown in the figure, the following layers and corresponding functions are provided:

[0077] The hardware and software layer 1060 includes hardware components and software components. Examples of hardware components include a mainframe 1061, RISC (Reduced Instruction Set Computer) architecture-based servers 1062, 1063, a blade server 1064, a storage device 1065, and network and networking components 1066. Depending on the embodiment, the software components may include network application server software 1067 and database software 1068.

[0078] The virtualization layer 1070 provides an abstraction layer that can give examples of virtual entities such as a virtual server 1071, virtual storage 1072, a virtual network 1073 including a virtual private network, a virtual application and operating system 1074, and a virtual client 1075.

[0079] In one embodiment, the management layer 1080 may provide the following functions: Resource provisioning 1081 dynamically procures computing and other resources used to perform tasks within the cloud computing environment. Metering and pricing 1082 tracks the costs of resources as they are used within the cloud computing environment and processes billing or invoicing for the consumption of those resources. In one embodiment, these resources may include application software licenses. Security verifies the identity of cloud consumers and tasks and protects data and other resources. User portal 1083 provides consumers and system administrators with access to the cloud computing environment. Service level management 1084 allocates and manages cloud computing resources to ensure that the required service levels are met. Service Level Agreement (SLA) planning and execution 1085 pre-arranges and procures cloud computing resources for which future demands are anticipated in accordance with the SLA.

[0080] The workload layer 1090 provides examples of capabilities that can utilize a cloud computing environment. Examples of workloads and capabilities that can be provided from this layer include mapping and navigation 1091, software development and lifecycle management 1092, virtual classroom education delivery 1093, data analysis processing 1094, transaction processing 1095, and machine learning model training (MLT) based on metrics in this subject, as described with reference to, for example, Figure 3, Figure 4, or Figure 5A 1096.

[0081] This subject may include the following items.

[0082] Item 1. Define metrics for evaluating the performance of machine learning (ML) models, This includes providing an ML engine, and the ML engine is It receives a set of training settings and a training dataset containing a first feature attribute and a first label attribute, The process involves generating at least one machine learning model using a training dataset, according to a set of training settings, wherein the machine learning model is configured to predict a first label attribute from a first feature attribute. To provide metric values ​​for the generated machine learning model, This involves retrieving the training settings and relevant metric values ​​from previous runs of the ML engine, The first label attribute receives a selection of desired values ​​for a metric to predict based on the current training dataset, Using the acquired set of training settings, determine the set of training settings that corresponds to the desired value of the metric, This involves controlling the ML engine to generate a machine learning model using the current training dataset, according to a determined set of training settings. A computer implementation method configured to deploy a generated machine learning model in order to predict the value of a first label attribute.

[0083] Item 2. Determining the set of training settings is Creating a training set that includes entries for each set of training settings from the acquired set of training settings, wherein each entry includes an input vector and an associated output vector, where the input vector includes metadata describing the metric values ​​and attributes, and the output vector includes the set of training settings. The method of item 1, comprising training a predictive model using a training set to predict a set of settings by providing a desired value for a metric and metadata of the attributes of the current training dataset as input to the predictive model in order to receive the set of settings determined above as output.

[0084] Item 3. The method of Item 1, which involves determining a set of settings, including selecting a set of settings from the acquired set of settings that corresponds to a desired value for the metric.

[0085] Item 4. Any method of Items 1 to 3, wherein the set of training settings includes a first subset of settings for preprocessing the training dataset, a second subset of settings for selecting one or more learning algorithms suitable for prediction, and a third subset of settings for training optimization.

[0086] Item 5. The metadata describing the attributes includes at least one of the following: the number of feature attributes, the type of feature attribute, and the type of label attribute, or the method of Item 2 or 4.

[0087] Item 6. Any method of items 1 through 5, further comprising normalizing metric values ​​to a range having a predefined granularity, wherein the acquired set of settings is provided according to the granularity, and as a result, each set of acquired settings represents the corresponding respective granularity level.

[0088] Item 7. The method of receiving a selection, including receiving a selection at the granular level, as described in Item 6.

[0089] Item 8. Receiving a selection involves displaying a graphical user interface that includes a slider, allowing the user to set a value by moving the slider's indicator, the slider containing a normalized range of values, and the set value of the slider being the selected and received value of the metric, in the manner of Item 6 or 7.

[0090] Item 9. Any method described in Items 1 through 8 above, where the metric describes the training runtime, the inference of the machine learning model, or both, the accuracy of the machine learning model, or a combination thereof.

[0091] Item 10. The metric is a combination of multiple metric types, each metric type relating to a separate set of settings from a set of settings, in any of the methods described in Items 1 through 9 above.

[0092] Item 11. Any method of items 1 through 10, further comprising recording the determined set of settings and the values ​​of the associated metrics.

[0093] The present invention may be a system, method, or computer program product or combination thereof at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or multiple mediums) storing computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0094] A computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. A computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random-access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures on which instructions are recorded, and any suitable combination thereof. As used herein, computer-readable storage media should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through optical fiber cables), or electrical signals transmitted through wires.

[0095] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and transfers those computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0096] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk(R) and C++, and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed as a standalone software package, either entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, to carry out aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute a computer-readable program instruction by personalizing the electronic circuit using state information of the computer-readable program instruction.

[0097] Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or both, illustrating methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block in the flowcharts or block diagrams, or both, and combinations of blocks in the flowcharts or block diagrams, or both, can be implemented by computer-readable program instructions.

[0098] These computer-readable program instructions can be supplied to a computer or other programmable data processing device processor to enable a machine to form means for implementing functions / operations specified in a flowchart or block diagram or both blocks, by having the instructions executed by the processor of the computer or other programmable data processing device. These computer-readable program instructions may be stored on a computer-readable storage medium on which the instructions are stored, which can be instructed to function in a particular manner to a computer, a programmable data processing device, or other device or a combination thereof, so that the storage medium on which the instructions are stored contains a product containing instructions that implements a mode of function / operation specified in a flowchart or block diagram or both blocks.

[0099] Computer-readable program instructions may be loaded into a computer, other programmable device, or other device in order to perform a series of operational steps on the computer, other programmable device, or other device to realize a computer implementation process, such that instructions executed on the computer, other programmable device, or other device implement the functions / operations specified in the flowchart or block diagram or both.

[0100] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in a block may be performed in an order different from the order shown in the diagram. For example, two consecutively shown blocks may, depending on the functions involved, actually be performed as one step, in parallel or substantially parallel, or partially or entirely overlapping in time, or the blocks may be executed in reverse order. It should also be noted that each block in a block diagram or flowchart or both, and combinations of blocks in a block diagram or flowchart or both, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or by implementing a combination of dedicated hardware and computer instructions.

Claims

1. A computer implementation method for training machine learning models, It receives a set of training settings and a training dataset containing a first feature attribute and a first label attribute, The process of generating at least one machine learning model based on the training dataset and the set of training settings, wherein the machine learning model is configured to predict the first label attribute from the first feature attribute, To provide metric values ​​for the generated machine learning model, Obtaining a set of training settings for one or more previous actions and the corresponding metric values, To receive a desired value for the metric for predicting the value of the first label attribute based on the current training dataset, Based on the acquired set of training settings, determine the set of training settings corresponding to the desired value of the metric, The process involves generating a machine learning model based on the determined set of corresponding training settings, The generated machine learning model is deployed. Computer implementation methods, including those mentioned above.

2. Determining the corresponding set of training settings is Creating a training set, wherein the training set is at least one entry for each set of training settings from the set of acquired training settings, each entry being an input vector and an associated output vector, the input vector being metadata describing the values ​​and attributes of the metric associated with the set of training settings, and the output vector being the set of training settings, To predict the corresponding set of training settings, a predictive model is trained based on the training set. The computer implementation method according to claim 1, including the method described in claim 1.

3. Determining the corresponding set of training settings is Selecting a set of training settings from the acquired set of training settings that corresponds to the desired value of the metric. The computer implementation method according to claim 1, including the method described in claim 1.

4. The computer implementation method according to claim 1, wherein the corresponding set of training settings includes a first subset of settings for preprocessing the training dataset, a second subset of settings for selecting one or more learning algorithms suitable for the prediction, and a third subset of settings for training optimization.

5. The metadata describing the attribute includes at least one of the following: the number of feature attributes, the type of the feature attribute, and the type of the label attribute; the computer implementation method according to claim 2.

6. The further includes normalizing the metric to a range of values ​​having a predefined granularity, wherein the acquired set of training settings is provided according to the granularity, and each set of the acquired set of training settings represents the corresponding respective granularity level. The computer implementation method according to claim 1.

7. To receive the desired value for the metric, The computer implementation method according to claim 6, further comprising receiving a selection of granularity levels.

8. To receive the desired value for the metric, The computer implementation method according to claim 6, further comprising displaying a graphical user interface, the graphical user interface including a slider that allows a user to set a value, the slider including a range of normalized values, and the set value of the slider being the desired value for the metric.

9. The metric corresponds to at least one of the following: the training runtime, the inference of the machine learning model, and the accuracy of the machine learning model; the computer implementation method according to claim 1.

10. The computer implementation method according to claim 1, wherein the metric is a combination of two or more metric types, and each metric type is associated with a separate set of training settings from the set of training settings.

11. Recording the determined set of corresponding training settings and the values ​​of the metrics associated therewith. The computer implementation method according to claim 1, further comprising:

12. A computer implementation method for defining metrics for evaluating the performance of a machine learning model, wherein the computer implementation method is It receives a set of training settings and a training dataset containing a first feature attribute and a first label attribute, Using the training dataset, generate at least one machine learning model according to the set of training settings, wherein the machine learning model is configured to predict the first label attribute from the first feature attribute. To provide metric values ​​for the generated machine learning model, Obtaining a set of training settings for previous runs of the ML engine and the corresponding metric values, Receiving a selection of desired values ​​for the metric to predict the value of the first label attribute based on the current training dataset, Using the acquired set of training settings, determine the set of training settings corresponding to the desired value of the metric. Using the current training dataset, the ML engine is controlled to generate a machine learning model according to the determined set of corresponding training settings. Deploying the generated machine learning model to predict the value of the first label attribute. Computer implementation methods, including those mentioned above.

13. Creating a training set which includes entries for each set of training settings from the set of training settings obtained, wherein each entry includes an input vector and an associated output vector, the input vector includes metadata describing the values ​​and attributes of the metric associated with the set of training settings, and the output vector includes the set of training settings, To receive the determined set of corresponding training settings as output, the predictive model is trained using the training set to predict the set of corresponding training settings by providing the desired value of the metric and the metadata of the attribute of the current training dataset as input to the predictive model. The computer implementation method according to claim 12, further comprising:

14. A computer system, wherein the computer system is Processor and Readable storage medium and The computer program instructions are stored in the readable storage medium, and the computer program instructions are transmitted to the processor. It receives a set of training settings and a training dataset containing a first feature attribute and a first label attribute, The process of generating at least one machine learning model based on the training dataset and the set of training settings, wherein the machine learning model is configured to predict the first label attribute from the first feature attribute, To provide metric values ​​for the generated machine learning model, Obtaining a set of training settings for one or more previous actions and the corresponding metric values, To receive a desired value for the metric for predicting the value of the first label attribute based on the current training dataset, Based on the acquired set of training settings, determine the set of training settings corresponding to the desired value of the metric, The process involves generating a machine learning model based on the determined set of corresponding training settings, The generated machine learning model is deployed. A computer system consisting of computer program instructions for performing a specific action.

15. Determining the corresponding set of training settings is Creating a training set, wherein the training set is at least one entry for each set of training settings from the set of acquired training settings, each entry being an input vector and an associated output vector, the input vector being metadata describing the values ​​and attributes of the metric associated with the set of training settings, and the output vector being the set of training settings, To predict the corresponding set of training settings, a predictive model is trained based on the training set. The computer system according to claim 14, including the following:

16. Determining the corresponding set of training settings is Selecting a set of training settings from the acquired set of training settings that corresponds to the desired value of the metric. The computer system according to claim 14, including the following:

17. The computer system according to claim 14, wherein the corresponding set of training settings includes a first subset of settings for preprocessing the training dataset, a second subset of settings for selecting one or more learning algorithms suitable for the prediction, and a third subset of settings for training optimization.

18. The metadata describing the attribute includes at least one of the following: the number of feature attributes, the type of the feature attribute, and the type of the label attribute; the computer system according to claim 15.

19. The invention further includes computer program instructions for normalizing the metric to a range of values ​​having a predefined granularity, wherein the acquired set of training settings is provided according to the granularity, and each set of the acquired set of training settings represents the corresponding respective granularity level. The computer system according to claim 14.

20. To receive the desired value for the metric, Receive a selection of granularity levels. The computer system according to claim 19, including the following:

Images