Identification and application of hyperparameters in machine learning

Abstract

Methods and systems for determining suitable hyperparameters for a machine learning model and / or feature engineering process are provided. Suitable machine learning models and associated hyperparameters are determined by analyzing a data set. Suitable hyperparameter values ​​for compatible machine learning models having one or more common hyperparameters and compatible data set architectures are identified. Hyperparameters can be ranked according to their respective impact on model performance metrics, and hyperparameter values ​​identified as having greater impact can be searched more aggressively.

Description

Background Art

[0001] Some machine learning algorithms may require hyperparameters, which affect how the algorithm performs. For example, hyperparameters can set the number of iterations, sample size, complexity, and can reflect assumptions about the machine learning model and training data. Feature engineering algorithms may also have hyperparameters, and may similarly affect how feature engineering is implemented. A data scientist can try to discover the optimal hyperparameters for a given machine learning algorithm and / or feature engineering algorithm based on heuristics and his or her experience, but this approach can be inconsistent and unreliable across different datasets, machine learning algorithms, and data scientists.

[0002] Hyperparameters can also be searched algorithmically using a brute force approach. A search algorithm can be executed to find the optimal hyperparameters in the set of all possible combinations, but this approach can require exponentially more computational time as the number of hyperparameters increases. To complicate matters, the search algorithm may require its own hyperparameters, and it may take a significant amount of time to tune these hyperparameters to obtain a usable search result. Summary of the invention

[0003] According to an embodiment of the disclosed subject matter, a computer-implemented system, medium, and method may include: receiving a first data set having a first data architecture; generating metadata based on attributes of the data set; selecting, by a computer processor, a machine learning model suitable for application to the data set based on the metadata; for each of a plurality of hyperparameters associated with the selected machine learning model, identifying the extent to which each hyperparameter affects one or more performance indicators of the selected machine learning model; identifying a first version of the selected machine learning model; obtaining a plurality of previously stored hyperparameter values ​​associated with the first version of the selected machine learning model based on the following steps: identifying a second version of the selected machine learning model, the second version having the same one or more hyperparameters as the first version of the selected machine learning model, and identifying similarities between the first data architecture and a second data architecture of a second data set associated with the second version of the selected machine learning model, and determining a value range for one or more of the previously stored hyperparameter values ​​based on a threshold. For each of the plurality of hyperparameters associated with the first version of the selected machine learning model that is the same as a hyperparameter in the one or more hyperparameters associated with the second version of the selected machine learning model, the method may include: selecting a first set of hyperparameter values ​​based on the identified degree of influence of each associated hyperparameter and from the determined value range; and for each of the plurality of hyperparameters associated with the first version of the selected machine learning model that is different from the hyperparameter in the one or more hyperparameters associated with the second version of the selected machine learning model, the method may include selecting a second set of hyperparameter values ​​based on the identified degree of influence of each associated hyperparameter. The method may also include training the first version of the selected machine learning model using the selected first set of hyperparameter values, the selected second set of hyperparameter values, and the data set. The metadata may include at least one selected from the following: the size of the training set, the shape of the data set, the number of features in the data set, the percentage of data field types in the data set, the type of classification problem, the difference in data field types in the data set, and an indication of whether the data set follows a statistical distribution. The method may further include executing an auxiliary machine learning model based on the metadata as input, the auxiliary machine learning model returning a selection of the first version of the selected machine learning model and returning appropriate machine learning hyperparameter values ​​for use by the first version of the selected machine learning model. The one or more performance indicators may include at least one selected from the group consisting of: accuracy, error rate, precision, recall, area under a receiver operating characteristic (ROC) curve, and area under a precision-recall curve.The method may further include executing an auxiliary machine learning model using the plurality of hyperparameters associated with the first version of the selected machine learning model as input, the auxiliary machine learning model returning a ranking of the plurality of hyperparameters according to the degree of influence on the one or more performance indicators of the first version of the selected machine learning model. The method may further include identifying a hyperparameter value for each of the plurality of hyperparameters based on a search, the search having a variable granularity, wherein the granularity of the search corresponds to the degree of influence of each of the plurality of hyperparameters on the one or more performance indicators of the first version of the selected machine learning model. The method may further include identifying a hyperparameter value within a determined range of values ​​for one or more of the hyperparameters of the first version of the selected machine learning model based on a search, the search having a variable granularity, wherein the granularity of the search corresponds to the degree of influence of each of the plurality of hyperparameters on the one or more performance indicators of the first version of the selected machine learning model. The method may further include identifying a hyperparameter value within a determined range of values ​​for one or more of the hyperparameters of the first version of the selected machine learning model based on a search, the search having a variable granularity, wherein the granularity of the search corresponds to the degree of influence of each of the plurality of hyperparameters on the one or more performance indicators of the first version of the selected machine learning model. The magnitude of the threshold may vary based on the extent to which the one or more previously stored hyperparameters affect one or more performance metrics of the first version of the selected machine learning model.

[0004] Additional features, advantages and embodiments of the disclosed subject matter may be set forth or understood by considering the following detailed description, drawings and claims. In addition, it should be understood that the preceding summary and the following detailed description are illustrative and intended to provide further explanation without limiting the scope of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The accompanying drawings are included to provide a further understanding of the disclosed subject matter, and are incorporated into and constitute a part of this specification. The accompanying drawings also illustrate embodiments of the disclosed subject matter and, together with the detailed description, serve to explain the principles of the embodiments of the disclosed subject matter. No attempt is made to show structural details in greater detail than is necessary for a basic understanding of the disclosed subject matter and the various ways in which it can be practiced.

[0006] Figure 1 An example method for determining optimal hyperparameters for a machine learning model according to an embodiment of the disclosed subject matter is shown.

[0007] Figure 2A An example method for determining optimal hyperparameters for a machine learning model according to an embodiment of the disclosed subject matter is shown.

[0008] Figure 2B An example method for determining optimal hyperparameters for a machine learning model according to an embodiment of the disclosed subject matter is shown.

[0009] Figure 3 A computing device according to an embodiment of the disclosed subject matter is shown.

[0010] Figure 4 A network configuration according to an embodiment of the disclosed subject matter is shown.

[0011] Figure 5 Example network and system configurations are shown in accordance with embodiments of the disclosed subject matter. DETAILED DESCRIPTION

[0012] Embodiments disclosed herein provide techniques for identifying parameters used in machine learning models based on repeatable techniques that can be effectively performed by automated, computerized systems. Suitable hyperparameters for a machine learning model can first be identified, for example, by examining the data on which the machine learning model will operate and comparing the data with previous data used in conjunction with other machine learning models. A suitable machine learning model can then be selected based on the similarity of the examined data to other data sets for which suitable machine learning models are known. Alternatively or additionally, the hyperparameters to be searched for when training the selected machine learning model can be identified based on the relative contribution of the hyperparameters to the performance of the model, as determined by one or more performance indicators associated with the model. Alternatively or additionally, the values ​​to be searched and / or the granularity of searching for individual hyperparameter values ​​can be identified using the automated and computerized techniques disclosed herein.

[0013] As used herein, the term "suitable" refers to a parameter or parameter value that achieves correct operation of a system (e.g., a machine learning system). A suitable value may be the least preferred value within a range of possible values, but still achieves correct operation of the system. Preferably, a suitable value can be said to achieve improved system operation when compared to another value within the range of possible values, but may not be the best possible value.

[0014] The term "algorithm" as used herein refers to both a single algorithm or multiple algorithms that may be used simultaneously or serially in a "stacked" fashion.

[0015] The term "model" as used herein refers to a machine learning algorithm and associated one or more suitable parameters and / or hyperparameters.

[0016] The machine learning system can allow data scientists to create machine learning models. Data scientists can collect one or more data sets from various sources (e.g., databases). Feature engineering algorithms can extract features of interest from the data sets. The feature engineering algorithm can then modify the extracted features, create new features, and remove features to create a new feature-engineered data set. The data scientist can then select a machine learning algorithm to create a model based on the feature engineering data set. This is also called training the model. The machine learning algorithm can be configured using one or more parameterized values ​​that specify how the machine learning algorithm will be executed, and these parameterized values ​​are called hyperparameters. In general, data scientists can develop customized indicators that may have priority importance in solving the problem at hand. Indicators may include, for example, accuracy, error rate, development time, precision, and recall. It is important to select hyperparameter values ​​so that the machine learning algorithm is executed as much as possible according to the needs of the data scientist. It should be understood that, as previously described, the feature engineering algorithm can also be configured using hyperparameters to similarly affect how it is executed.

[0017] This subject matter discloses an automated and computer-based method for identifying hyperparameters and applying them to machine learning and / or feature engineering algorithms. Several embodiments are disclosed, which can be used individually, jointly, or in any combination thereof. Similarly, the processes employed in each embodiment can be performed simultaneously, asynchronously, or in an order different from that shown and described.

[0018] In one embodiment, the disclosed method may include receiving a data set and generating metadata based on attributes of the data set. The metadata may then be used to identify a suitable machine learning model and suitable hyperparameter values. The identified machine learning model may then be configured to be trained using the identified suitable hyperparameters and the received data set.

[0019] In one embodiment, the disclosed method can select a machine learning model and train the one or more models using one or more data sets. From one or more subsequently trained models, one or more hyperparameters that have a greater impact on the behavior of the model on the one or more data sets can be identified and compiled into a list. For each hyperparameter on the list, a range of values ​​can be searched to identify suitable values ​​that cause the machine learning model to perform according to the performance indicators specified by the data scientist. The selected machine learning model can then be configured to be trained using the identified suitable hyperparameter values.

[0020] In one embodiment, the disclosed method may select a machine learning model and a data set. The data set may be arranged according to an architecture. The method may receive version data associated with the selected machine learning model. The method may identify previously used hyperparameter values ​​for the machine learning model corresponding to the selected machine learning model based on one or both of the version data associated with the previously used hyperparameter values ​​and the data set architecture. Based on the previously used hyperparameter values, a range of values ​​may be searched within a threshold range to identify suitable values ​​that cause the machine learning model to perform according to a performance metric specified by a data scientist. The selected machine learning model may then be configured to be trained using the identified suitable hyperparameter values.

[0021] Figure 1 An example flowchart 100 of a method for selecting a suitable machine learning model and associated hyperparameters based on one or more data sets is shown. In 105, the system obtains one or more data sets, for example, by receiving the data sets from the system, for example, in response to the selection of the data set by a data scientist or other user. The data set can be, for example, a tenant data set containing customer data and subject to privacy and security protocols. Therefore, based on the permission level, the user of the machine learning system (e.g., a data scientist or computer engineer) can be restricted from viewing some or all of the data contained in the one or more data sets received in 105. The data sets received in stage 105 can be combined and randomly split to create a training set and a hold-out set. The training set can be used to subsequently train the selected machine learning model in stage 120, and the hold-out set can be used to evaluate the accuracy of the selected machine learning model. In 110, metadata describing the properties of the data set received in 105 can be generated, and the metadata can be based on the data set, other data available to the system, and data entered by the system user. The metadata can be generated based on all data sets jointly, or it can be generated based on each data set. The metadata can be generated by a separate data set preprocessing stage, or generated in conjunction with another machine learning process, as described in further detail herein. The metadata may include data describing, for example, the size and shape of the dataset, the number of fields in the dataset, the percentage breakdown of field types in the dataset (e.g., categorical, numeric, text), the type of classification problem, dataset variance, whether there is a correlation between the data and labels, whether the dataset follows a statistical distribution, etc. After the metadata is generated in stage 110, the metadata may be saved in a database or other data structure according to conventional methods.

[0022] In stage 115, a suitable machine learning model can be selected from a plurality of machine learning models based at least on the metadata generated in 110. The machine learning model can be selected in part based on its known advantages, and one machine learning model can be selected over another based on the content of the data set and the metadata describing it. For example, if the metadata reveals that the data set contains a large proportion of classified data, a machine learning model known to perform well on classified data can be selected. Stage 115 can be performed by an auxiliary machine learning model. The auxiliary machine learning model can accept one or more data sets and associated metadata, and based on the one or more data sets and metadata, return the selected machine learning model and the appropriate hyperparameter values ​​of the hyperparameters associated with the selected machine learning model. It should be understood that the hyperparameter values ​​can be numeric or non-numeric. The auxiliary machine learning model can operate according to any conventional machine learning algorithm, such as grid search, random search, Bayesian methods, etc. In 120, the appropriate machine learning model selected in 115 can be trained using the selected appropriate hyperparameter values ​​and the data set received in 105.

[0023] Figure 2A An example flow chart 200 for selecting one or more suitable values ​​for a machine learning model hyperparameter is shown. At 205, the method receives a selection of a machine learning model and one or more data sets. The machine learning model may be selected according to the method 100 via the auxiliary machine learning model in stage 115, selected by a user, or selected according to other conventional methods known in the art. The machine learning model selected in 205 may have been previously trained across multiple data sets, and may have generated data useful for determining the degree of influence of each hyperparameter associated with the selected machine learning model. The performance indicators may be determined automatically or by a data scientist, and may include, for example, accuracy, error rate, precision, recall, area under the precision-recall curve (AuPR), area under the receiver operating characteristic curve (AuROC), etc. It should be understood that the selection of one or more performance indicators may be related to evaluating whether one hyperparameter value is better than another, and no hyperparameter value may perform better than all other hyperparameter values ​​in terms of each performance indicator.

[0024] In stage 210, method 200 may identify and rank hyperparameters associated with the machine learning model selected in stage 205 according to their respective effects on one or more performance indicators. This may be accomplished using an auxiliary machine learning model that receives the previously discussed data obtained by training the selected machine learning model on multiple data sets and one or more selected performance indicators, and returns a ranking of the associated hyperparameters according to their respective effects on the one or more selected performance indicators. The auxiliary machine learning model may utilize a random forest algorithm or other conventional machine learning algorithms capable of calculating the importance of hyperparameters in a model.

[0025] Having identified and arranged the hyperparameters according to their impact in stage 210, stage 215 may use any conventional machine learning algorithm to search for suitable hyperparameter values. Preferably, a grid search algorithm may be used, which allows the specification and / or granularity of the search to be specified for each hyperparameter. Suitable values ​​may be searched for hyperparameters that are determined to have a stronger impact on the performance indicator at a larger granularity. Suitable values ​​may be searched for hyperparameters that are determined to have a weaker impact on the performance indicator at a smaller granularity. In this way, by allocating search time, computing resources may be more efficiently utilized, and search results may be more productive at this time. For example, for hyperparameters that are determined to have a strong impact, 50 possible hyperparameter values ​​may be checked, while for hyperparameters that have a weak impact, 5 hyperparameter values ​​may be checked. The search process 215 may then return one or more hyperparameter values ​​for each hyperparameter associated with the machine learning algorithm selected in stage 205.

[0026] In stage 220, the hyperparameter values ​​determined in stage 215 may be stored in a hyperparameter memory, which may be implemented using any conventional storage device. The hyperparameter memory may be indexed by the model and include data describing, for example, the time and date the model was trained, the code version of the algorithm used by the model, the architecture of the dataset on which the model was trained, the performance of the model according to the performance metrics previously discussed, the value of each hyperparameter of the model, etc. Future hyperparameter selections may be accelerated by using the hyperparameter memory to look up suitable hyperparameter values ​​in which matching data may be found, rather than performing each of steps 210-215. In stage 225, the machine learning model selected in 205 may be trained using the dataset selected in 205 and the selected suitable hyperparameter values ​​determined in stage 215.

[0027] Figure 2BAn example flowchart 250 for selecting one or more suitable values ​​for a machine learning model hyperparameter is shown. At 255, the method receives a selection of a machine learning model and one or more data sets. The machine learning model may be selected according to method 100 via an auxiliary machine learning model in stage 115, selected by a user, or selected according to other conventional methods known in the art. The machine learning model selected in stage 255 may have an associated version that may be identified in 260. For example, the version may correspond to a version of a machine learning algorithm employed by the model. Newer versions of the machine learning algorithm may utilize new hyperparameters that were missing from previous versions and / or may have eliminated other hyperparameters. In general, in multiple versions of the machine learning algorithm, all or most of the hyperparameters may remain the same, thereby ensuring advantage by storing and recalling previously used suitable hyperparameters in a hyperparameter memory.

[0028] At stage 265, method 250 may retrieve hyperparameters and their associated values ​​previously used with the selected machine learning model from the previously described hyperparameter memory. The retrieved hyperparameters and their associated values ​​may have been used in the same or different versions of the selected machine learning model. As previously discussed with respect to stage 220, versions of machine learning algorithms may be stored in a hyperparameter memory. The hyperparameter memory may also be associated with the architecture of the dataset for which the model was trained. Because the dataset may affect the applicability of the hyperparameters, stage 265 may also compare the architecture of the dataset selected in 255 with the architecture of the dataset stored in the hyperparameter memory to evaluate similarities and differences.

[0029] At stage 270, a hyperparameter value for each previously used hyperparameter that was determined to be the same as the hyperparameter of the selected machine learning model version can be searched based on the threshold. For example, if the previously used hyperparameter value was 10, stage 270 can select a threshold range of 5 to test the suitability of values ​​between 5 and 15. As previously described, any conventional machine learning algorithm can be used to search for suitable hyperparameter values. Preferably, a grid search or equivalent algorithm can be used that allows the specification and / or granularity of the search to be specified for each hyperparameter.

[0030] In stage 275, suitable values ​​can be searched for hyperparameters determined to have a stronger impact on the performance indicator at a larger granularity. Suitable values ​​can be searched for hyperparameters determined to have a weaker impact on the performance indicator at a smaller granularity. In this way, by allocating search time, computing resources can be used more efficiently, and the search results may be more productive at this time. For example, for hyperparameters determined to have a strong impact, 50 possible hyperparameter values ​​between 5 and 15 can be checked, while for hyperparameters with a weak impact, 5 hyperparameter values ​​between 5 and 15 can be checked. As previously mentioned, it should be understood that hyperparameter values ​​can be digital or non-digital. In addition to the granularity of the search, the size of the threshold can vary based on the similarity between the version of the selected machine learning model and the version of the machine learning model for which the previously used hyperparameters are available in the hyperparameter memory. The similarity can be determined based on, for example, the number of data fields of the data set architecture received in 255, which matches the data fields of the data set architecture associated with the previously used hyperparameter values. Alternatively, or in addition, similarity can be determined based on the number of hyperparameters that are identical or different from hyperparameters of a machine learning model version associated with a previously used hyperparameter value. In the case where the similarity is identical or substantially similar, the threshold can be selected to be smaller in size, while in the case where similarity is lacking, the threshold can be selected to be larger in size. In this way, more hyperparameter values ​​can be tested for suitability in situations where it may be less certain whether those previously used hyperparameter values ​​are suitable for current use. In stage 280, the machine learning model selected in 255 can be trained using the data set selected in 255 and the selected suitable hyperparameter values ​​determined in stage 275.

[0031] As previously mentioned, the various embodiments disclosed herein may be used individually, in combination, or in any combination thereof. For example, given one or more selected data sets, methods 100, 200, and 250 may be used in combination to reduce the overall computational effort required to determine a suitable machine learning model and suitable hyperparameters. In this example, method 100 may be used to select a suitable machine learning model and suitable hyperparameter values ​​for a given data set. The hyperparameters associated with the hyperparameter values ​​determined in method 100 may be applied to Figure 2A and Figure 2B One or both of. Figure 2BIn the example of method 100, the associated values ​​of the hyperparameters identified in method 100 can be obtained from a hyperparameter store in which previously used hyperparameter values ​​exist. The threshold size can be set based on the similarity between the dataset architecture and the machine learning model hyperparameters. For hyperparameters of the selected machine learning model that are the same as previously used hyperparameters, the hyperparameter values ​​can be searched within a range defined by the threshold size of the previously used hyperparameter values ​​and at a granularity defined by their impact as performed in method 200. For hyperparameters that are different from previously used hyperparameters, hyperparameter values ​​can be searched at a granularity defined by their impact as performed in method 200. The suitable machine learning model determined in stage 115 of method 100 can then be trained using the dataset and hyperparameter values ​​selected in stage 105, the hyperparameter values ​​being selected based on their impact on the performance metrics in stage 215 of method 200 and the similarity of the dataset architecture and machine learning model versions stored in the hyperparameter store as determined in stage 275 of method 250.

[0032] The embodiments disclosed herein may allow for more efficient selection of appropriate hyperparameters for machine learning models and feature engineering than using traditional techniques. For example, the disclosed embodiments may determine appropriate machine learning models and associated hyperparameters more efficiently than comparable traditional machine learning techniques, and / or may use fewer computing resources than using traditional techniques. This is due to the use of the techniques disclosed herein, which reduce the computational time involved by reducing the size of the search space when determining appropriate machine hyperparameters for a given purpose, thereby providing an increase in efficiency without loss of generality or accuracy. In addition, the embodiments disclosed herein may overcome related shortcomings that occur in traditional multi-tenant frameworks.

[0033] Embodiments of the presently disclosed subject matter can be implemented in and used with a variety of components and network architectures. Figure 3 2 is an example computing device 20 suitable for implementing embodiments of the presently disclosed subject matter. The device 20 may be, for example, a desktop or laptop computer, or a mobile computing device such as a smart phone, tablet computer, etc. The device 20 may include a bus 21 that interconnects the main components of the computer 20, such as a central processor 24, a memory 27, such as a random access memory (RAM), a read-only memory (ROM), flash RAM, etc., a user display 22, such as a display screen, a user input interface 26, which may include one or more controllers and associated user input devices, such as a keyboard, a mouse, a touch screen, etc., a fixed storage 23 such as a hard drive, flash memory, etc., a removable media component 25 operable to control and receive an optical disk, a flash drive, etc., and a network interface 29 operable to communicate with one or more remote devices via a suitable network connection.

[0034] The bus 21 allows data communication between the central processor 24 and one or more memory components, which, as previously described, may include RAM, ROM, and other memories. Typically, the RAM is the main memory into which the operating system and application programs are loaded. The ROM or flash memory components may contain a basic input and output system (BIOS), which controls basic hardware operations, such as interaction with peripheral components. Application programs resident in the computer 20 are typically stored on and accessed through a computer-readable medium, such as a hard drive (e.g., fixed memory 23), an optical drive, a floppy disk, or other storage medium.

[0035] Fixed storage 23 may be integrated with computer 20, or may be separate and accessible through other interfaces. Network interface 29 may provide a direct connection to a remote server through a wired or wireless connection. Network interface 29 may provide such a connection using any suitable technology and protocol readily understood by those skilled in the art, including digital cellular telephone, WiFi, Bluetooth(R), near field, etc. For example, network interface 29 may allow the computer to communicate with other computers via one or more local, wide area or other communication networks, as described in further detail below.

[0036] Many other devices or components (not shown) may be connected in a similar manner (e.g., document scanners, digital cameras, etc.). Figure 3 All components shown in the figure are present to practice the present disclosure. Components may be interconnected in a manner different from that shown. Figure 3 The operation of the illustrated computer is well known in the art and will not be discussed in detail in this application. The code implementing the present disclosure may be stored in a computer readable storage medium, such as one or more of memory 27, fixed storage 23, removable media 25, or in a remote storage location.

[0037] Figure 4An example network arrangement according to an embodiment of the disclosed subject matter is shown. One or more devices 10, 11, such as local computers, smart phones, tablet computing devices, etc., can be connected to other devices via one or more networks 7. Each device can be a computing device as described above. The network can be a local area network, a wide area network, the Internet, or any other suitable communication network, and can be implemented on any suitable platform including wired and / or wireless networks. These devices can communicate with one or more remote devices, such as a server 13 and / or a database 15. The remote device can be directly accessed by the device 10, 11, or one or more other devices can provide intermediate access, such as a server 13 provides access to resources stored in a database 15. Devices 10, 11 can also access a remote platform 17 or services provided by the remote platform 17, such as cloud computing arrangements and services. The remote platform 17 can include one or more servers 13 and / or databases 15.

[0038] Figure 5 An example arrangement of an embodiment according to the disclosed subject matter is shown. One or more devices or systems 10, 11, such as a remote service or service provider 11, a user device 10, such as a local computer, a smart phone, a tablet computing device, etc., can be connected to other devices via one or more networks 7. The network can be a local area network, a wide area network, the Internet, or any other suitable communication network, and can be implemented on any suitable platform including a wired and / or wireless network. Devices 10, 11 can communicate with one or more remote computer systems, such as a processing unit 14, a database 15, and a user interface system 13. In some cases, devices 10, 11 can communicate with a user-oriented interface system 13, which can provide access to one or more other systems, such as a database 15, a processing unit 14, etc. For example, a user interface 13 can be a user-accessible web page that provides data from one or more other computer systems. The user interface 13 can provide different interfaces to different clients, such as providing a human-readable web page to a web browser client on the user device 10, and providing a computer-readable application programming interface or other interface to the remote service client 11.

[0039] The user interface 13, database 15 and / or processing unit 14 may be part of an integrated system or may include multiple computer systems communicating via a private network, the Internet or any other suitable network. One or more processing units 14 may be, for example, part of a distributed system, such as a cloud-based computing system, a search engine, a content delivery system, etc., which may also include or communicate with a database 15 and / or user interface 13. In some arrangements, the analysis system 5 may provide back-end processing, such as pre-processing of stored or acquired data by the analysis system 5 before transmitting it to the processing unit 14, database 15 and / or user interface 13. For example, the machine learning system 5 may provide various predictive models, data analysis, etc. to one or more other systems 13, 14, 15.

[0040] More generally, various embodiments of the presently disclosed subject matter may include or be embodied in the form of computer-implemented processes and apparatus for practicing these processes. Embodiments may also be implemented in the form of a computer program product having a computer program code containing instructions, the instructions being contained in a non-transitory and / or tangible medium, such as a floppy disk, a CD-ROM, a hard drive, a USB (Universal Serial Bus) drive, or any other machine-readable storage medium, such that when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter. Embodiments may also be implemented in the form of computer program code, for example, whether stored in a storage medium, loaded into and / or executed by a computer, or transmitted through some transmission medium, such as by wire or cable, by optical fiber, or by electromagnetic radiation, such that when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing embodiments of the disclosed subject matter. When implemented on a general-purpose microprocessor, the computer program code segments configure the microprocessor to create specific logic circuits.

[0041] In some configurations, a set of computer-readable instructions stored on a computer-readable storage medium can be implemented by a general-purpose processor, which can convert a general-purpose processor or a device containing a general-purpose processor into a special-purpose device configured to implement or execute instructions. The embodiment can be implemented using hardware, which can include a processor, such as a general-purpose microprocessor and / or an application-specific integrated circuit (ASIC), which implements all or part of the techniques of the embodiments according to the disclosed subject matter in hardware and / or firmware. The processor can be connected to a memory, such as a RAM, ROM, flash memory, a hard disk, or any other device capable of storing electronic information. The memory can store instructions suitable for execution by the processor to perform the techniques of the embodiments according to the disclosed subject matter.

[0042] For the purpose of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or limit the embodiments of the disclosed subject matter to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. These embodiments are selected and described in order to explain the principles of the embodiments of the disclosed subject matter and their practical applications, so that other persons skilled in the art can utilize these embodiments and various embodiments with various modifications that can be suitable for the intended specific use.

Claims

1. A computer-implemented method performed in an automated machine learning system, the method comprising: receiving a first tenant data set containing customer data within a multi-tenant framework and subject to privacy and security protocols, the first tenant data set having a first data schema; generating metadata based on the attributes of the first tenant dataset; selecting, by a computer processor, a machine learning model suitable for application to the first tenant dataset based on the metadata; For each hyperparameter of a plurality of hyperparameters associated with the selected machine learning model, identifying a degree of influence of each hyperparameter on one or more performance metrics of the selected machine learning model; identifying a first version of the selected machine learning model; A plurality of previously stored hyperparameter values ​​associated with the first version of the selected machine learning model are obtained based on the following steps: identifying a second version of the selected machine learning model, the second version having the same one or more hyperparameters as the first version of the selected machine learning model, and identifying similarities between the first data schema and a second data schema of a second tenant dataset within the multi-tenant framework containing customer data and subject to privacy and security protocols, the second tenant dataset being associated with a second version of the selected machine learning model; determining a value range for one or more of the previously stored hyperparameter values ​​based on the threshold value; For each hyperparameter in the plurality of hyperparameters associated with the first version of the selected machine learning model that is the same as a hyperparameter in the one or more hyperparameters associated with the second version of the selected machine learning model: selecting a first set of hyperparameter values ​​based on the identified degree of influence of each associated hyperparameter and from the determined range of values; as well as For each hyperparameter in the plurality of hyperparameters associated with the first version of the selected machine learning model that is different from a hyperparameter in the one or more hyperparameters associated with the second version of the selected machine learning model: selecting a second set of hyperparameter values ​​based on the identified degree of influence of each associated hyperparameter; as well as A first version of the selected machine learning model is trained using the selected first set of hyperparameter values, the selected second set of hyperparameter values, and the first tenant dataset.

2. The method according to claim 1, wherein the metadata comprises at least one item selected from the following: the size of the training set, the number of features in the first tenant dataset, the percentage of data field types in the first tenant dataset, the type of classification problem, the diversity of data field types in the first tenant dataset, and an indication of whether features in the first tenant dataset follow a statistical distribution.

3. The method of claim 1, wherein selecting a machine learning model comprises: An auxiliary machine learning model is executed based on the metadata as input, and the auxiliary machine learning model returns a selection of a first version of the selected machine learning model and returns appropriate machine learning hyperparameter values ​​for use by the first version of the selected machine learning model.

4. The method of claim 1, wherein the one or more performance indicators include at least one selected from the group consisting of accuracy, error rate, precision, recall, area under a receiver operating characteristic (ROC) curve, and area under a precision-recall curve.

5. The method of claim 1, wherein identifying the degree of impact further comprises: An auxiliary machine learning model is executed using the multiple hyperparameters associated with the first version of the selected machine learning model as input, and the auxiliary machine learning model returns a ranking of the multiple hyperparameters based on their impact on the one or more performance indicators of the first version of the selected machine learning model.

6. The method of claim 1 , wherein selecting based on the degree of impact identified for each associated hyperparameter further comprises: A hyperparameter value is identified for each of the plurality of hyperparameters based on a search having a variable granularity, wherein the granularity of the search corresponds to an extent to which each of the plurality of hyperparameters affects the one or more performance metrics of the first version of the selected machine learning model.

7. The method of claim 1, wherein the selecting a first set of hyperparameter values ​​based on the identified degree of influence of each associated hyperparameter and from the determined value range further comprises: Identifying a hyperparameter value based on a search within a determined range of values ​​for one or more of the hyperparameters of the first version of the selected machine learning model, wherein the search has a variable granularity, wherein the granularity of the search corresponds to a degree of influence of each of the multiple hyperparameters on one or more performance metrics of the first version of the selected machine learning model.

8. The method of claim 1, wherein the selecting a second set of hyperparameter values ​​based on the identified degree of influence of each associated hyperparameter further comprises: Identifying a hyperparameter value based on a search within a determined range of values ​​for one or more of the hyperparameters of the first version of the selected machine learning model, wherein the search has a variable granularity, wherein the granularity of the search corresponds to a degree of influence of each of the multiple hyperparameters on one or more performance metrics of the first version of the selected machine learning model.

9. The method of claim 1 , wherein the magnitude of the threshold varies based on the degree to which the one or more previously stored hyperparameters affect one or more performance metrics of the first version of the selected machine learning model.

Images