COMPUTER IMPLEMENTED METHODS, COMPUTING SYSTEMS, AND COMPUTER PROGRAMS FOR SELECTING A SUITABLE MACHINE LEARNING PIPELINE FOR PROCESSING A NEW DATA SET (WITH APPLICATIONS IN AUTOMATED ARTIFICIAL INTELLIGENCE ITEM RECOMMENDATION)

Abstract

To solve the problem that processing for selecting a suitable model for a different application becomes more important if it is assumed that a demand for an ML application increases to increase the number of available ML models.SOLUTION: A plurality of rating values corresponding to respective performance of a plurality of pipelines about one data set among a plurality of data sets are stored. A distance between pairs of each data set among the plurality of data sets is determined in a latent space. The plurality of data sets are clustered in the latent space, and a representative data set in each cluster is identified. A suitable pipeline set is selected from among a plurality of sets according to a set of rating values corresponding to the respective performance of a pipeline selected for the representative data set in response to the reception of a new data set. The new data set is processed in the suitable pipeline set.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates generally to item recommendation and has particular application in the field of automated artificial intelligence (AutoAI). Computer-implemented methods for selecting a suitable machine learning pipeline for processing a dataset are provided, along with systems and computer program products implementing such methods. Corresponding methods also find application in network-based item recommender systems. [Background technology]

[0002] Machine learning (ML) models are widely used in science and technology, with applications in computer vision, speech recognition, image processing, medical diagnostics, genetic analysis, drug design, network fault analysis, and intruder detection, among many other examples. Machine learning techniques involve processing a dataset of training data from a real-world application in order to train or optimize a model for the application in question in relation to a base model for that application. This model can then be applied to perform inference tasks based on new (previously unseen) data generated in the application. Many machine learning models have been developed, including various models based on artificial neural networks, such as so-called "deep learning" networks, support vector machines, Bayesian networks, decision tree models, and many others. Given that the demand for ML applications is growing and the number of available ML models is increasing, the process of selecting the appropriate model for different applications is becoming more and more important. The AutoAI system aims to assist with this task, applying intelligent automation to the machine learning pipeline selection and tuning process, from preparing and selecting training datasets, to training and evaluating models, and tuning model parameters. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "Finite-time Analysis of the Multiarmed Bandit Problem", Auer et al., Machine Learning 47(2): 235-256 [Non-Patent Document 2] "Wisdom of the Better Few: Cold Start Recommendation via Representative based Rating Elicitation", Nathan Lui et al., RecSys'11, 2011 [Non-Patent Document 3] "Probabilistic matrix factorization for automated machine learning", Fusi et al., Advances in Neural Information Processing Systems, 2018 [Non-Patent Document 4] "Tapas: Train-less Accuracy Predictor for Architecture Search", Istrate et al., Proceedings of the AAAI Conference on Artificial Intelligence, 33:3927-3934, 2019 Summary of the Invention [Problem to be solved by the invention]

[0004] Given the growing demand for ML applications and the increasing number of available ML models, the process of selecting the right model for different applications is becoming increasingly important. The AutoAI system aims to assist with this task, applying intelligent automation to the selection and tuning process of machine learning pipelines, from preparing and selecting training datasets, to training and evaluating models, and tuning model parameters. [Means for solving the problem]

[0005] Certain aspects of the present disclosure provide a computer-implemented method for selecting a suitable machine learning pipeline for processing a new dataset. The method includes storing rating values ​​for a number of machine learning pipelines and a number of N datasets previously processed by the pipelines, each rating value providing a rating to the performance of the pipeline on a dataset. For each pair u in the previously processed datasets, where i and j are i=1 to N, j=1 to N, i≠j, the number of machine learning pipelines is selected. i , u j On the other hand, this method uses u in the latent space for the dataset. i From j Distance d to i,j Determine the distance d i,j is the dataset u j The pipeline selected in a predetermined manner based on the rating value for the dataset u i , which corresponds to the predicted value of regret that occurs when a rating is given for the dataset u iThe rating value for the performance of that pipeline on the representative dataset is a monotonically decreasing function of the rating value for that pipeline's performance on the representative dataset. The method then clusters the previously processed datasets in the latent space described above, and identifies in each cluster a representative dataset for which the distance of that dataset from other datasets in the cluster is minimized across the cluster. The method further includes, in response to receiving a new dataset, selecting a set of preferred pipelines to process the dataset, each preferred pipeline being selected according to the rating value for the representative dataset. The new dataset is then processed in this set of preferred pipelines.

[0006] A method for implementing this aspect of the disclosure uses a latent space in which datasets are embedded to select a suitable pipeline for a new dataset based on the notion of "pairwise expected regrets" that determine the distance between each pair of datasets in the latent space and capture the expected drop in the performance rating of the pipeline for one dataset when the pipeline is selected based on its performance rating for another dataset. These embodiments are further based on the idea that all datasets, including the new one, are drawn from a multimodal probability distribution in the latent space. By clustering the datasets in this space, the datasets can be grouped into "neighborhoods" for which a number of representative datasets can be identified, one from each neighborhood. The new dataset is likely to be near these representative datasets in the latent space. By selecting a suitable pipeline based on the rating values ​​of the representative datasets, the selected pipeline is likely to be given a high rating for the new dataset. The pipelines can be selected to maximize this probability, as described in more detail below. In this way, methods embodying the present disclosure provide an effective and elegant solution to the problem of selecting a suitable ML pipeline for a new dataset in AutoAI. This leads to more efficient training of ML models, better performing models, and improved behavior of user applications against these models. Aside from handling the cold start problem, embodiments of the present disclosure also provide improved control of the exploitation vs. exploration tradeoff. This and other advantages are described in more detail below.

[0007] Typically, the new dataset is a sample dataset provided by a user that broadly represents the user data on which the ML model is trained for inference on the user application. An embodiment may be envisioned in which only one preferred pipeline is selected for a new dataset. The method described above then selects a pipeline according to the ratings on the representative dataset, which can be trained by processing the new dataset and then applied for inference on new data for the user application. However, typically, a user has a budget that allows training of multiple pipelines before choosing one for the user application. An embodiment of the present disclosure may then select multiple preferred pipelines according to the ratings on multiple representative datasets. This allows evaluation of a pipeline selected based on ratings on more than one representative dataset, one of which is closest to the new dataset in the latent space described above.

[0008] Dataset u i Pipeline P that has been given a rating k The regret for the dataset u i The maximum rating value for the pipeline performance on the dataset u i Pipeline P k Regret can conveniently be determined as a function of the difference between the rating value for the performance of pipeline P for a given dataset compared to the highest rated pipeline. k This depends on the decline in ratings for

[0009] The above-mentioned predetermined manner for selecting a pipeline (on which the predicted regret, and therefore the calculation of the pairwise distance, is based) may vary in different embodiments. In some embodiments, a predetermined pipeline may be selected, such as the pipeline with the highest rating, in which case the predicted value of regret will be the regret that will occur for that pipeline. However, in a preferred embodiment, a probabilistic process is used such that the predetermined manner for selecting a pipeline based on the rating value for the dataset includes selecting the pipeline as a random variable from a probability distribution conditioned on the rating value for the dataset. The predicted value of regret is then calculated based on the number of pipelines u j The dataset is selected based on the rating value for i The pipeline rating includes a probability-weighted average of regrets that occur for a pipeline that is given a rating for . This provides the basis for a controlled search of potential pipeline recommendations for new datasets.

[0010] A particularly preferred method allows for approximation of the representative dataset that is closest to the new dataset in the latent dataset space. In particular, for a number of preferred pipelines selected according to rating values ​​for the corresponding representative datasets, these embodiments determine a rating value for the performance of each preferred pipeline for the new dataset and calculate regrets resulting for each preferred pipeline for the new dataset. At least one further preferred pipeline is then selected based on the rating value for the representative dataset corresponding to the preferred pipeline with the lowest regret for the new dataset, in the manner described above. The new dataset is then processed in each further preferred pipeline. Calculating regrets for the preferred pipelines that are rated for the new dataset allows the representative dataset that is closest to the new dataset in the latent space to be determined with increasing accuracy as more pipelines are rated for the new dataset. Additional pipelines can then be probabilistically selected by drawing from a probability distribution for this closest representative dataset. This approach, as will be further described below, allows for a controlled search for pipeline recommendations (i.e., selected pipelines), while the selection of the pipeline is still guided by the ratings for the representative datasets that are close to the new dataset in the latent dataset space. The pipeline selected based on the rating of the closest representative dataset is most likely to produce the lowest regrets on the new dataset, which, due to the monotonicity of regrets in relation to the pipeline's rating, produces the highest predicted rating for the performance of the selected pipeline on the new dataset.

[0011] The probability of selecting a pipeline from a probability distribution for a dataset may vary in direct proportion to the rating value for that pipeline's performance on that dataset. Thus, pipelines that are given higher ratings are more likely to be selected. In particularly preferred embodiments, the probability distribution for a dataset depends on a parameter β, whose value determines the dependence of the probability distribution on the rating values ​​for that dataset. Preferred methods select the value of parameter β to provide the desired dependence of the probability distribution on the rating values. In these embodiments, this "greedy parameter" β provides an efficient control of the exploitation vs. exploration tradeoff, while still allowing the search to be guided by the ratings on the representative dataset.

[0012] Another aspect of the present disclosure provides a computing system for selecting a suitable machine learning pipeline for processing a new dataset. The system includes a memory for storing the rating values ​​described above and embedding logic configured to determine pairwise distances in a latent space for the dataset. The computing system also includes clustering logic for clustering the dataset in the latent space to identify a representative dataset as described above. Pipeline selector logic of the system selects a set of suitable pipelines as described above and feeds the new dataset to the set of pipelines for processing. The computing system may include a machine learning system implementing multiple machine learning pipelines.

[0013] A further aspect of the present disclosure provides a computer program product including a computer-readable storage medium embodied with program instructions executable by a computing system to cause the computing system to perform the pipeline selection method described above.

[0014] While the present disclosure finds particular application in the field of AutoAI, methods embodying the present disclosure may also be applied with advantage in item recommender systems. Thus, a further aspect of the present disclosure provides a computer-implemented method for selecting an item for recommendation to a new user in a recommender system implemented in a computer network. The method includes storing, for a plurality of items and a plurality of N previous users of the system, rating values ​​each indicating a previous user's rating for an item. For each pair (u, j, i=1 to N, j=1 to N, i≠j) among the previous users, a rating value is stored. j , u j ), this method uses u in the latent space for the user i From j Distance d to i,j Here, the distance d i,j is the user u j The items selected in a predetermined manner based on the rating value of i The regret of an item corresponds to the predicted value of the regret that occurs when a user u gives a rating. i The representative user is a monotonically decreasing function of the rating values ​​for the items rated by. The method then clusters the previous users in the latent space and identifies in each cluster a representative user whose distance to the user from other users in the cluster is minimized across the cluster. The method further includes, in response to detection of the new user by the recommender system, selecting a set of preferred items to recommend to the user, where each preferred item is selected according to the rating value for the representative user. The recommendation of the set of preferred items is provided for display to the new user via a user interface.

[0015] A preferred embodiment may now receive a rating value for each preferred item from the new user and calculate the regret incurred for each preferred item rated by the new user. At least one further preferred item for recommendation may then be probabilistically selected (in a manner similar to the probabilistic pipeline selection process described above) based on the rating values ​​for the representative user corresponding to the preferred items having the lowest regret for the new user.

[0016] Embodiments of the present disclosure are explained in more detail below, by way of illustrative and non-limiting examples, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0017] [Figure 1] 1 is a schematic diagram of a computing system for implementing methods embodying the present disclosure. [Diagram 2] FIG. 1 is a diagram of component modules of a computing system for selecting a suitable ML pipeline for a new dataset. [Diagram 3] 3 illustrates steps of a pipeline selection method performed by the system of FIG. 2. [Figure 4] 1 is a schematic illustration of regrets calculated in a pipeline selection method embodying the present disclosure; [Diagram 5] 1 is a schematic illustration of the distribution of a dataset in a latent space used in a method embodying the present disclosure. [Figure 6] 6 is an illustration of the clustering of the dataset in the latent space of FIG. [Figure 7] FIG. 3 illustrates steps of a preferred pipeline selection method in an embodiment of the system of FIG. 2. [Figure 8a] 13 is an illustration of the effect of parameter β on pipeline selection probability in a preferred embodiment; [Figure 8b]13 is an illustration of the effect of parameter β on pipeline selection probability in a preferred embodiment; [Figure 9] FIG. 13 illustrates a comparison of the performance of a pipeline selection method embodying the present disclosure with a method based on item recommendations. [Figure 10] 1 is a schematic illustration of a network-based recommender system. [Figure 11] 11 illustrates method steps embodying the present disclosure for recommending items in the system of FIG. 10. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The present disclosure may be a system, method, or computer program product, or any combination thereof. The computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform aspects of the present disclosure.

[0019] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves on which instructions are recorded, and any suitable combination of the above. As used herein, computer-readable storage media should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0020] The computer readable program instructions described herein may be downloaded from a computer readable storage medium into each computing / processing device or may be downloaded to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage.

[0021] The computer readable program instructions for carrying out the operations of the present disclosure may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, or similar, and traditional procedural programming languages ​​such as the "C" programming language or similar. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or remote server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by using state information of the computer readable program instructions to individually configure the electronic circuitry to perform aspects of the present disclosure.

[0022] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer readable program instructions.

[0023] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to cause a machine to create means for implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, and these computer readable program instructions may be stored in a computer readable storage medium capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored constitutes an article of manufacture containing instructions that implement aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0024] Also, the computer readable program instructions can be loaded into a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable data processing apparatus, or other device, implement the functions / operations identified in one or more blocks of the flowcharts and / or block diagrams, and a series of operational steps can be executed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process.

[0025] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions described in the blocks may occur in a different order than the order described in the drawings. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of multiple blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that executes the specified functions or operations or executes a combination of dedicated hardware instructions and computer instructions.

[0026] The described embodiments can be implemented as a computer-implemented method for selecting a suitable machine learning pipeline for processing a new data set or for selecting recommended items for a new user in a network-based recommender system. Such a method can be implemented by a computing system including one or more general-purpose or special-purpose computers providing functionality for implementing the operations described herein, each of which can include one or more machines (real or virtual). The steps of the method for carrying out the present disclosure can be implemented by program instructions, e.g., program modules, that are implemented by a processor of the system. In general, program modules can include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computing system can be implemented in a distributed computing environment, such as a cloud computing environment, where tasks are performed by remote processing devices linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer system storage media, including memory storage devices.

[0027] An embodiment of the present disclosure considers an approach to determine which particular ML pipeline performs best for a given application. This can theoretically be determined by processing a training dataset in all available pipelines for that application and then selecting the best performing pipeline. However, training ML models is a computationally and time-intensive task. Users typically provide a sample dataset that represents their application data for testing purposes, but may only have the budget to test a limited number of pipeline options. The performance of the final trained model in the user application, and therefore the performance of the application itself, depends on the best ML pipeline selected by the AutoAI process.

[0028] The problem of selecting a pipeline for a user dataset has some similarity to the problem of recommending items in recommender systems. Recommender systems use a variety of techniques to recommend items (products, movies, songs, etc.) to a user based on ratings given by other users on those items and / or ratings given by the user on other items. These systems rely on the key assumption that users who have given similar ratings on items in the past tend to give similar ratings to items in the future. Common approaches to recommender systems include collaborative filtering, content-based recommendation, and low-rank matrix factorization. However, these approaches all suffer from the so-called “cold start” problem, which arises when recommending items to new users for whom no previously observed rating information exists.

[0029] One way that recommender systems deal with the cold start problem is to recommend the most popular items to new users. Such an approach makes the most of the ratings available for a user, but does not explore potentially better alternative items. The so-called “epsilon-greedy” approach (see “Finite-time Analysis of the Multiarmed Bandit Problem”, Auer et al., Machine Learning 47(2): 235-256) addresses this problem by recommending the most popular items with probability (1-ε) and a random item with probability ε. This epsilon-greedy approach explicitly controls the trade-off between exploitation and exploration via the parameter ε, but the exploration of alternative items (alternatives to the most popular item) is random and unguided. Matrix-based factorization techniques have also been proposed, where the ratings for a large number of items are reconstructed by a linear combination of the ratings for only a small number of items. Then, a new user only needs to rate a few items to infer ratings for the remaining items (see, for example, "Wisdom of the Better Few: Cold Start Recommendation via Representative based Rating Elicitation", Nathan Lui et al., RecSys'11, 2011). With matrix-based factorization, there is a risk of exposing the user to some poorly rated items before making recommendations. Moreover, these methods do not explicitly control the exploitation-exploration tradeoff, and the factorization problem is difficult to solve due to the discontinuity of objectives.

[0030] Item recommendation based methods have been applied to ML pipeline selection in "Probabilistic matrix factorization for automated machine learning", Fusi et al., Advances in Neural Information Processing Systems, 2018. This method uses matrix-based factorization as discussed above and does not address the cold start problem beyond that. "Tapas: Train-less Accuracy Predictor for Architecture Search", Istrate et al., Proceedings of the AAAI Conference on Artificial Intelligence, 33:3927-3934, 2019 proposes a method to predict the accuracy of deep learning networks on unseen datasets. This method requires characterizing the new dataset based on the features of various datasets and encoding the internal structure of the deep learning model. This encoding process is critical, especially for any ML pipeline.

[0031] Improved methods for selecting ML pipelines for new user datasets, and for dealing generally with the cold-start problem, would be highly desirable.

[0032] 1 is a block diagram of an exemplary computing device for implementing methods embodying the present disclosure. The computing device is shown in the form of a general purpose computer 1. Components of computer 1 may include a processing device, such as one or more processors, represented by processing unit 2, a system memory 3, and a bus 4 that couples various system components, including system memory 3, to processing unit 2.

[0033] Bus 4 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures, including, by way of example and not limitation, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0034] The computer 1 typically includes a variety of computer-readable media. Such media may be any available media accessible by the computer 1, including volatile and non-volatile media, and removable and non-removable media. For example, the system memory 3 may include computer-readable media in the form of volatile memory, such as random access memory (RAM) 5 and / or cache memory 6. The computer 1 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 7 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (commonly referred to as a "hard drive"). Also, not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such an example, each may be connected to the bus 4 by one or more data medium interfaces.

[0035] The memory 3 may include at least one program product having one or more program modules configured to perform the functions of the embodiments of the present disclosure. By way of example, a program / utility 8 having a set (at least one) of program modules 9 may be stored in the memory 3, as well as an operating system, one or more application programs, other program modules, and program data. The operating system, one or more application programs, other program modules, and program data, or a combination thereof, may each include an implementation of a networking environment. The program modules 9 generally perform the functions and / or methods of the embodiments of the present disclosure described herein.

[0036] The computer 1 may also communicate with one or more external devices 10, such as a keyboard, pointing device, display 11, one or more devices that allow a user to interact with the computer 1, or any device (e.g., network card, modem, etc.) that allows the computer 1 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur through an input / output (I / O) interface 12. The computer 1 may also communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, through a network adapter 13. As shown, the network adapter 13 communicates with other components of the computer 1 through a bus 4. The computer 1 may also communicate with additional processing devices 14, such as one or more GPUs (graphics processing units), FPGAs, or integrated circuits (ICs), to implement embodiments of the present disclosure. It should be understood that other hardware and / or software components may be used with the computer 1. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0037] FIG. 2 illustrates in schematic form the component modules of an exemplary computing system for selecting an ML pipeline in an AutoAI system. The computing system 20 comprises a memory 21 and control logic, generally indicated at 22, which is functional for selecting a suitable ML pipeline for a new user dataset. The control logic 22 comprises an embedding module 23, a clustering module 24 and a pipeline selector 25. As indicated by the dashed lines in the drawing, in a preferred embodiment, the pipeline selector 25 includes a performance evaluator 26. Each of the logic modules 23-26 is functional for implementing a particular step of the pipeline selection method described below. In particular, the system 20 comprises a set of logic modules P, for k=1 to M, where P k The machine learning pipeline P k Each pipeline includes functionality for training and evaluating an ML model on an input dataset. As illustrated generally for pipeline P1, each pipeline includes a training module 28 for training an ML model 30 and an inference module 29 for evaluating the performance of the ML model 30. The ML model 30 may be implemented in any desired manner, but may include one or more component models of one or more types. Each pipeline P kmay include additional functionality (not shown) of a commonly known type, for example, preparing (cleaning or otherwise pre-processing) the dataset and tuning pipeline parameters. When the dataset 32 ​​is processed by the ML system 27 and the ML pipeline is trained for a user application, the parameters of the trained model are stored in a model database 33 for subsequent use in the user application. In general, the ML system 27 may be local or remote to the computing system 20, but in some embodiments may be integrated with the system 20.

[0038] The modules 23-26 of the control logic 22 interface with a memory 21 which stores various data structures used in the operation of the system 20. These data structures each represent a pipeline P for a previously processed data set 32. k The rating value {R j,k} set 34 and the distance {d i,j The data set includes a set of {{dataset_name}} 35, clustering data defining dataset clusters 36, and a representative dataset 37 derived from the dataset clusters 36. The preferred embodiment also stores a set of performance ratings 38 determined for a new user dataset 39, as described below.

[0039] In general, the functionality of logic 23-30 may be implemented by software (e.g., program modules) or hardware or a combination thereof. The described functionality may be distributed differently among the system modules in other embodiments, and functionality of one or more modules may be combined. The component modules of computing system 20 may be provided in one or more computers of the computing system. For example, all modules may be provided in computer 1, or the modules may be provided in one or more computers / servers of the AutoAI system to which a user may connect via a network for input of data sets to be processed. Such networks may generally include one or more component networks and / or interconnections, including the Internet. System memory 21 may be implemented by one or more memory / storage components associated with one or more computers of system 20.

[0040] In operation, system 20 selects a suitable set of ML pipelines for processing a new dataset 39 in ML system 27. The basic steps performed by system 20 are shown in Figure 3. Step 40 selects a set of suitable ML pipelines for processing a new dataset 39 in ML system 27. j,k} to the system memory 21. In particular, for k=1 to M, j=1 to N, there are M pipelines P k and a number of N previously processed u j For a data set 32 ​​denoted by j,k is the dataset u j Pipeline P kAs illustrated, the ML system 27 may provide a large number (e.g., hundreds or thousands) of pipeline implementations. These may be trained over time on a similar number of data sets, both test data sets and user data sets, although training is typically a continuing process as new training data becomes available. Pipeline performance is given a rating herein in any convenient manner. The rating value may, for example, indicate the overall accuracy of the inference results produced by the ML model, or may be based on precision or recall or both, or some combination of these or other performance metrics or both.

[0041] In step 41, the embedding module 23 determines a set of pairwise distances for the previously processed dataset in a conceptual latent space, denoted χ for the dataset. In particular, for each pair u in the previously processed dataset, i and j, i=1 to N, j=1 to N, i≠j, i , u j The embedding module 23 embeds u in the latent space χ i From j Distance d to i,j Determine the distance d i,j is the dataset u j A pipeline is selected in a predetermined manner based on the rating value for the dataset u i u corresponds to the predicted value of regret that occurs when a rating is given for the dataset u i The resulting pairwise distance is asymmetric, i.e., d i,j and d j,iThe specific manner in which a pipeline may be selected based on the rating value for a dataset, and the calculation of regret values ​​herein, are described further below. The resulting distance {d i,j} is stored in the system memory 21 at 35.

[0042] At step 42, the clustering module 24 clusters the previously processed datasets 32 in the latent space χ. Clustering is a well-known process for assigning items (herein, datasets) to groups (clusters) such that the distance (measured in some defined space) between the items inside each cluster is minimized. Here, therefore, the clustering module 24 clusters the datasets in the latent space χ such that the distance between pairs of datasets (and therefore the predicted regret between pairs) inside each cluster is minimized. Data defining the resulting dataset clusters are stored in the system memory 21 at 36. At step 43, the clustering module uses the distance set 35 to identify a representative dataset in each cluster. The representative dataset is a dataset that is determined in each cluster by the distance d from the other datasets in that cluster to that dataset. i,j The representative data set is then identified as the data set for which χ is minimized across that cluster. The representative data set is therefore located near the mode of the data set distribution in the latent space χ. The representative data set identified by this process is stored in system memory 21 at 37.

[0043] Receipt of a new dataset 39 by the system 20 for processing by the ML system 27 is represented by step 44 in FIG. 3. In step 45, the pipeline selector 25 selects a set of preferred ML pipelines for processing the new dataset. Here, each preferred pipeline is selected as a function of a rating value for the representative dataset. This selection process is described in more detail below. In step 46, the new dataset 39 is provided to the ML system 27 for processing by each preferred pipeline selected by the pipeline selector 25.

[0044] The above process embeds the processed datasets in a latent space χ based on the notion of pairwise expected regret, where the performance rating of a pipeline on a dataset reflects the degradation that occurs when that pipeline is selected based on its rating on another dataset. The datasets are clustered into groups or "neighborhoods" and representative datasets are identified from the neighborhood that are close to the modes of the dataset distribution. A new dataset 39 is likely to be close to one of the modes and therefore close to the representative dataset. By selecting a suitable pipeline based on its rating on the representative dataset, the selected pipeline is likely to have a high rating and therefore perform well on the new dataset.

[0045] A scenario may be envisaged in which only one pipeline can be trained for the new dataset, in which case the pipeline selector 25 selects the preferred pipeline in step 45 according to its rating for one of the representative datasets, e.g. as the pipeline with the highest rating for that dataset. However, it is common that multiple preferred pipelines are selected in step 45 according to their ratings for the representative datasets. This increases the probability of selecting the high-performing pipeline for the new dataset, since one of the representative datasets is closest to the new dataset in the latent space χ and thus produces the smallest predicted regret. Pipelining typically does not only include training, but also testing the performance of the trained model, e.g. on a holdout set, i.e. a subset of the original dataset that has been set aside to test inference performance. Based on the results of these performance tests, the pipeline selector 25 can identify the best performing pipeline for the user application.

[0046] In cases where a user's budget allows for testing multiple pipelines, preferred embodiments employ particularly advantageous techniques for pipeline selection that allow for approximation of the representative dataset that is closest to the new dataset in the latent space χ and controlled search for pipeline recommendations. Exemplary embodiments that employ these techniques are described in detail below.

[0047] The preferred embodiment utilizes a probabilistic pipeline selection process. In particular, the above-mentioned predetermined manner for selecting a pipeline (on which the computation of predicted regrets and pairwise distances in step 41 of FIG. 3 is based) is a probabilistic process. In particular, a pipeline is selected as a random variable from a probability distribution conditioned on the rating values ​​for that dataset. The predicted values ​​of regrets are then calculated for the dataset u. j The dataset is selected based on the rating value for i In this embodiment, the dataset u j The dataset u is selected based on the ratings for i A rating is given to the pipeline P k The regret for the dataset u i The maximum rating value for the pipeline performance on the dataset u i Pipeline P k In particular, for the dataset u i Pipeline P that was given a rating for k Regret for G(k;u i )teeth,

number

[0048] Herein, to ensure that pipelines with higher ratings are more likely to be selected, the probability of selecting a pipeline from a probability distribution for a dataset varies directly with the rating value for that pipeline's performance on that dataset. In this example, for dataset u, j A pipeline P is constructed from a probability distribution conditioned on the rating value for k The probability of selecting P(k|u j ) is P(k|u j ) ∝exp(-β(M j -R j,k )) is satisfied, except that M j is the dataset u j This is the maximum rating of the pipeline's performance against R j,k is the dataset u j Pipeline P k It is a rating of the performance of the

[0049] Figure 5 is a simplified diagram illustrating the embedding of a dataset in the latent space χ. i From the dataset u j Distance d to i,j and the data set u j The probability distribution P(k|u j ) is a random variable from the dataset u i The expected regret value when a rating is given for

number

number

number

[0050] For ease of illustration in Fig. 5, the distances between the data sets are symmetric (d i,j =d j,i ), but in reality, these pairwise distances are asymmetric. Rating R i,k Regret G(k;u i ), the monotonicity also applies to the prediction. That is, the predicted regret value

number

[0051] Figure 6 is a simple illustration of the clustering process in this embodiment. The datasets are assumed to be drawn from a multimodal probability distribution in the latent space χ and, as illustrated, are clustered to minimize the distance between the datasets in each cluster, i.e., to minimize the predicted regret. Inside each cluster, a representative dataset (here u, shown in bold in the figure) is selected. r1 , u r2 , and ur3 A representative dataset is one whose distance from other datasets in the cluster to that dataset is minimized across the cluster. The representative dataset lies closest to the modes of the multimodal probability distribution and thus locally minimizes the expected pairwise regrets within the cluster. In this embodiment, affinity propagation is used for clustering and identifying the representative dataset, while respecting the asymmetry of pairwise distances. However, in other embodiments, any clustering method that respects asymmetric pairwise distances may be used.

[0052] FIG. 7 illustrates the steps of the pipeline selection process in this embodiment. Step 50 represents the arrival of a new data set 39 with a budget allocated for testing multiple pipelines. The previous operations correspond to steps 40-43 of FIG. 3 described above. In step 51, the pipeline selector 25 selects multiple preferred pipelines. The preferred pipelines are selected depending on their rating values ​​for the corresponding representative data set. In this example, the preferred pipelines are the pipelines that have been given the highest ratings for the corresponding representative data set. The number of pipelines selected depends on the allocated budget, as will be described below. In step 52, the new data set is processed in each of the preferred pipelines to determine a rating value for the performance of each preferred pipeline on the new data set. The resulting performance ratings 38 are stored in the memory 21, as shown in step 53. In step 54, the pipeline selector 25 selects multiple suitable pipelines for the new data set u. new The regrets G(k;u new In step 55, the pipeline selector calculates the lowest regret G(k;u) for the new data set. new ) a representative dataset u corresponding to a suitable pipeline r*In step 56, the pipeline selector 25 selects this representative data set u r* and selecting at least one further preferred pipeline based on a rating value for each of the representative datasets u. r* The probability distribution P(k|u r* ) are selected as random variables from the set of u new A rating 38 for the performance of each additional pipeline with respect to the new data set is determined and stored in step 58. In step 59, the pipeline selector identifies the optimal pipeline for the new data set as the pipeline having the highest rating value among the ratings (e.g., performance ratings) with respect to the new data set 38. The optimal pipeline parameters may then be stored in the database 33 for a user application in step 60.

[0053] If the dataset is sampled from a multimodal distribution, the new dataset is likely to be close to the modes of the distribution and therefore close to the representative dataset. new In the latent space χ, u new A particular representative data set u that is likely to be near r* The probability distribution P(k|u r* ) allows a pipeline to be recommended. Ideally, r* In the latent space χ, u new in the nearest neighborhood of, i.e.,

number

number

[0054] u r* Since the decision strictly requires that all pipelines be rated on new datasets, the above process r* We approximate by considering only the mode of the representative distribution as follows:

number

number

number

[0055] The above process provides an elegant way to ensure that the pipeline most likely to produce low regret values ​​and therefore high ratings is selected for a new dataset. Furthermore, the above-mentioned probability distribution P(k|u j ) allows us to control the trade-off between exploitation and exploration. This “greedy parameter” β is a function of the probability distribution P(k|u j ) is the dataset u j β determines the extent to which the probability distribution depends on the rating value for β. Figures 8a and 8b illustrate how adjusting the parameter β affects the probability distribution. Figure 8a shows the effect of adjusting the parameter β on the probability distribution for the data set u as β approaches infinity. j Rating R for j,k In relation to the distribution P(k|u j) in the extreme case, only the highest rated pipeline has a nonzero probability of being selected, leading to maximum utilization (greedy). Figure 8b shows the distribution P(k|u j ), where every pipeline has a nonzero probability of selection, and the probability is rated R j,k (exploitation and exploration). Decreasing the value of β makes the distribution more flat, until β=0 and all pipelines are equally likely to be selected. In this extreme case, the pipeline selection probability increases with the rating R j,k (maximum exploration). The greedy parameter β thus allows explicit control of the exploitation-exploration tradeoff. The value of β is j,k P(k|u j ) can be conveniently selected to provide a desired degree of dependency, allowing for a controlled exploration of potential pipelines while still being guided by the representative user ratings through the selection process for the preferred pipeline.

[0056] Embodiments may be envisaged in which the value of β can be set based on the requirements for the new data set, such as the budget. For example, a larger search may be appropriate, allowing for testing of many pipelines, whereas on the other hand, a smaller budget may limit the search. The embedding and clustering processes are computationally fast, and in some embodiments, recalculations may be performed for this purpose. However, in preferred embodiments, the optimal value of β may be determined based on the previously processed data set 32 ​​as the value that gives the best results for pipeline selection. In these embodiments, the performance evaluator 26 may select the optimal value of the parameter β by cross-validation with the previously processed data set. Cross-validation is a well-known technique. Essentially, in this specification, the performance evaluator 26 selects a test subset of the previously processed data set, which is treated as a "new" data set for testing the performance of the prediction process. By iteratively testing the performance for multiple test data sets and multiple values ​​of β, the optimal value may be easily determined. The performance evaluator may periodically recalculate this value as more data sets are consumed.

[0057] It will be appreciated that the above-described embodiments provide a very effective solution to the cold start problem and the automatic selection of the optimal ML pipeline for a new dataset. The described pairwise β-greedy cold start approach can provide exceptionally high performing ML models and enhanced inference results for user applications. This approach has been shown to outperform the epsilon-greedy approach with various parameter settings. As an example, the pairwise β-greedy and ε-greedy methods were trained with a budget of 5 pipelines, with the performance of about 5000 pipelines evaluated on about 500 datasets. Figure 9 plots the results for regrets calculated on the holdout set for various values ​​of the parameter ε. The regrets for the ε-greedy method were highest when ε=0 (maximum utilization, i.e., the most popular item (here, the pipeline with the highest rating) is always selected), and the regrets generally decreased as the search increased (ε became higher). Even when utilization is maximized (β approaches infinity), the pairwise β-greedy method yielded significantly lower regrets than those obtained for all values ​​of ε.

[0058] Of course, various alternatives and modifications can be made to the described embodiment. By way of example, various probability distributions can be assumed for rating-based pipeline selection in embodiments where the process is probabilistic.

[0059] While the above-described techniques are particularly useful in the AutoAI application described, embodiments of the present disclosure can also be applied to the cold start problem in item recommender systems. FIG. 10 is a schematic diagram of such a recommender system implemented in a computer network. The recommender system 70 is implemented in a server 71 to which a user computer 72 can connect via a network, shown generally at 73 herein. The network 73 can include one or more component networks and / or interconnections, including the Internet. The user computers 72 can include various types of user devices, such as desktop, laptop or tablet computers, mobile phones, PDAs (personal digital assistants), televisions, game consoles, etc., through which users can connect to the remote server 71. In a typical scenario, the recommender system 70 may be implemented in a server 71 (or a collection of servers) that provides a web-based application, such as a search engine, an online shopping site, a product comparison site, a movie / music / game download site, or any other service that users can access over a network.

[0060] In operating such a service, the recommender system 70 may recommend various items (e.g., products, songs, games, movies, software applications, etc.) to users based on user ratings of those items. These items are referred to herein as items. k Item identifiers denoted by (k=1,2,3,...) and data k The recommender system 70 has a set of information for these items, denoted by {item k data k}. In this specification, jA user identified by the user identifier denoted by (j=1,2,3,...) is k When you give a rating to an item with j,k This rating, denoted by u, is then recorded in a ratings database 75. This database is then collated with all users u who have previously given a rating to the item. j For item rating {R j,k} is stored in the system.

[0061] When the recommender system 70 detects new users with previously unseen user identifiers, it can use the cold start approach described above to recommend items to these users. This process is similar to the pipeline selection process; that is, items can be selected for recommendation to new users in the same way that a suitable pipeline is selected to process a new data set. k is the pipeline P k Corresponding to user u j is the dataset u j The recommender system 70 can thus be implemented in a manner similar to the system 20 of FIG. 2 with the embedding module 23, the clustering module 24, and the pipeline selector 25. Similarly, the memory 21 of the recommender system 20 stores the rating database (75 in FIG. 10) and the set of distances d i,j The database may also store user cluster data 36, ​​a collection of representative data sets 37, and may also store ratings 38 for new users.

[0062] The operation of the recommender system 70 is illustrated in FIG. k Previous users {u j Rating value for {R j,kIn step 81, the embedding module 23 determines a set of pairwise distances of previous users in the latent space χ for the user. In particular, for each pair u of N previous users, i and j, i=1 to N, j=1 to N, i≠j, i , u j On the other hand, the embedding module 23 embeds u i From j Distance d to i,j Determine the distance d i,j In a given manner, the user u j The items selected based on the rating value for user u i , which corresponds to the predicted value of regret that occurs when a user u i is a monotonically decreasing function of the rating value for the given item. The resulting distance {d i,j} is stored in the system memory 21 at 35.

[0063] At step 82, the clustering module 24 clusters the previous users in the latent space χ. As before, the clusters are defined in the usual way such that the distance between pairs of users (herein, and hence the expected regret between pairs) is minimized within each cluster. The resulting user clusters are stored in the system memory 21 at 36. At step 83, the clustering module uses the set of distances 35 to identify a representative user in each cluster. A representative user is a user who is in each cluster based on the distance d i,j The representative user is identified as the user for which χ is minimized over its cluster. The representative user is therefore located near the mode of the user distribution in the latent space χ. The representative user is stored in the system memory 21 at 37.

[0064] Detection of a new user by the recommender system 70 is represented in step 84 of FIG. 11. In step 85, the item selector 25 selects a set of suitable items to recommend to the new user. Each suitable item is selected according to its rating value for the representative user, e.g., as the item with the highest rating for that user. In step 86, the recommender system 70 provides a recommendation of the set of suitable items for display to the new user via a user interface at the user computer 72. Here, the system 70 selects item information data for the suitable items. k from database 74 and sends this information to the user computer for display. The item information, which may be integrated with other information by system 70, for example in a web page, is displayed to the user by server 71. The item information (e.g., photo and / or text with links for further information / item ratings) is thus displayed to the new user via user computer 72.

[0065] In a preferred embodiment of the present invention, this process can be extended with steps 87 to 91 of FIG. 11. Step 87 here represents the recommender system 70 receiving ratings from new users for each preferred item. In steps 88 to 90, the recommender system then selects at least one further preferred item in the same manner as further pipelines are selected in steps 54 to 56 of FIG. 7. That is, in step 88, the recommender system calculates regrets for each recommended item based on the received user ratings. The representative user u corresponding to the item with the lowest regret is selected. r* is identified in step 89. Next, in step 90, the further preferred item (or items) are sent to the representative user u r*Each further suitable item can be selected based on its rating value for the representative user u, exactly like the suitable pipeline selection process described above. r* The probability distribution P(k|u r* ) may be selected. In step 91, the recommender system then provides recommendations of each further suitable item for display to the new user via a user interface of the user computer 72.

[0066] In general, the implementation details for the pipeline selection embodiment described above can be similarly applied to item recommendations in the system 70. The β-greedy cold-start approach can thus be applied to item recommendations in a similar manner. Thus, the system 70 can recommend items to new users that are popular within a neighborhood (cluster) of previous users using a guided search of possible item recommendations via the representative user's ratings and the greedy parameter β. This provides a recommender system in which items recommended to new users will have high predicted ratings for those users.

[0067] In general, where a feature is described herein with reference to a pipeline selection method embodying the present disclosure, a corresponding feature in an item recommendation method embodying the present disclosure may be provided. Further, where a feature is described herein with reference to any method embodying the present disclosure, a corresponding feature in a system / computer program product embodying the present disclosure may be provided, and vice versa.

[0068] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements in comparison with the technology found in the market, or to enable other skilled in the art to understand the embodiments disclosed herein.

[0069] [Explanation of symbols] 2 Processing unit, 3 Memory, 5 RAM, 6 Cache, 7 Storage system, 10 External device, 11 Display, 12 I / O interface, 13 Network adapter, 14 GPA / FPGA / IC, 21 System memory, 22 Control logic, 23 Embedding module, 24 Clustering module, 25 Pipeline selector, 26 Performance evaluator, 27 ML system, 28 Training module, 29 Inference module, 30 ML model, 32 Processed dataset, 33 User model, 34 Rating {R j,k}, 35 distance {d i,j}, 36 Dataset clusters, 37 Representative datasets, 38 Ratings for new datasets, 39 New datasets, 70 Recommender systems, 71 Servers, 72 User computers, 73 Networks

Claims

1. 1. A computer-implemented method for selecting a suitable machine learning pipeline for processing a new dataset, comprising: storing a plurality of rating values ​​for a plurality of machine learning pipelines and a plurality of N datasets previously processed by the plurality of machine learning pipelines, each rating value corresponding to performance of a pipeline in the plurality of machine learning pipelines and for a dataset in the plurality of N datasets; For each pair u in the plurality of N previously processed data sets, i and j, i=1 to N, j=1 to N, i≠j, ｉ , u ｊ On the other hand, u in the latent space ｉ From ｊ Distance d to ｉ，ｊ determining the distance d ｉ，ｊ Let, be the dataset u ｊ The pipeline selected in a predetermined manner based on a set of rating values ​​for the data set u ｉ , and the regret for the selected pipeline corresponds to a predicted value of regret that occurs when a rating is given to the performance of the selected pipeline with respect to the dataset u ｉ the determining step comprising a monotonically decreasing function of the rating value with respect to the performance of the pipeline with respect to clustering the plurality of N previously processed data sets in the latent space and identifying in each cluster a representative data set for which each distance to the data set from other data sets in the cluster is minimized across the cluster; in response to receiving a new dataset, selecting a set of preferred pipelines from the plurality of machine learning pipelines for processing the new dataset, each preferred pipeline being selected according to a set of rating values ​​for the representative dataset; processing said new data set in said set of preferred pipelines; A method comprising:

2. The data set u ｊ The predetermined manner for selecting the pipeline based on the rating value for the dataset u ｊ as a random variable from a probability distribution conditioned on the rating value for the predicted value of the regret comprises a probability weighted average of a set of regrets occurring for each set of suitable pipelines; The method of claim 1.

3. determining a rating value for the performance of each pipeline in the set of preferred pipelines on the new dataset; calculating regrets incurred for each pipeline in the set of preferred pipelines for the new dataset; selecting at least one further suitable pipeline in the predetermined manner based on the rating value for the performance and the representative data set corresponding to the at least one further suitable pipeline having the lowest regret with respect to the new data set; processing said new data set in each of said at least one further suitable pipeline; The method of claim 2 , further comprising:

4. determining a rating value for the performance of each of the at least one further preferred pipeline on the new data set; responsive to said determination of the rating value for the performance of each of said at least one further suitable pipeline with respect to said new data set, identifying among said at least one further pipeline a pipeline having a highest rating value with respect to said new data set; The method of claim 3 further comprising:

5. The method of claim 4 , wherein the set of preferred pipelines comprises a set of pipelines that have been given the highest ratings on the corresponding representative data sets.

6. The data set u ｊ The probability of selecting the pipeline from the probability distribution over the data set u ｊ 6. The method of claim 2, wherein the rating value varies in direct proportion to the performance of the selected pipeline with respect to

7. The data set u ｊ the probability distribution for depends on a parameter β having a value that determines the dependence of the probability distribution on the rating value for the data set; The method further comprises selecting a value for the parameter β to provide a desired dependence of the probability distribution on the set of rating values ​​of the representative data set.

7. The method according to any one of claims 2 to 6.

8. The method of claim 7 , further comprising selecting an optimal value for the parameter β by cross-validation with the plurality of N data sets.

9. The data set u ｊ A pipeline P ｋ The probability of selecting P(k|u ｊ )teeth, P(k|u ｊ )∝exp(-β(M ｊ -R ｊ，ｋ )) It satisfies Here, M ｊ is the dataset u ｊ is the maximum rating of the pipeline's performance with respect to R ｊ，ｋ is the dataset u ｊ Pipeline P ｋ The rating for the performance of 9. The method according to claim 7 or 8.

10. The data set u ｉ Pipeline P that was given a rating for ｋ The regret for the data set u ｉ The maximum rating value for the pipeline performance on the dataset u ｉ The pipeline P ｋ The method according to any one of claims 1 to 9, wherein the rating is a function of the difference between the rating value for the performance of the player and the rating value for the performance of the player.

11. 1. A computing system for selecting a suitable machine learning pipeline for processing a new dataset, comprising: a memory subsystem containing program instructions; a processor in communication with the memory subsystem; wherein the program instructions cause the processor to: storing a plurality of rating values ​​for a plurality of machine learning pipelines and a plurality of N datasets previously processed by the plurality of machine learning pipelines, each rating value corresponding to performance of a pipeline in the plurality of machine learning pipelines and for a dataset in the plurality of N datasets; For each pair u in the plurality of N previously processed data sets, i and j, i=1 to N, j=1 to N, i≠j, ｉ , u ｊ On the other hand, u in the latent space ｉ From ｊ Distance d to ｉ，ｊ determining the distance d ｉ，ｊ Let, be the dataset u ｊ The pipeline selected in a predetermined manner based on a set of rating values ​​for the data set u ｉ , and the regret for the selected pipeline corresponds to a predicted value of regret that occurs when a rating is given to the performance of the selected pipeline with respect to the dataset u ｉ the determining includes a monotonically decreasing function of the rating value for the performance of the pipeline with respect to clustering the plurality of N previously processed data sets in the latent space and identifying in each cluster a representative data set for which each distance to the data set from other data sets in the cluster is minimized across the cluster; in response to receiving a new dataset, selecting a set of preferred pipelines from the plurality of machine learning pipelines for processing the new dataset, each preferred pipeline being selected according to a set of rating values ​​for the representative dataset; processing said new data set in said set of suitable pipelines; A computing system that performs the steps of:

12. The data set u ｊ The predetermined manner for selecting the pipeline based on the rating value for the dataset u ｊ as a random variable from a probability distribution conditioned on the rating value for the predicted value of the regret comprises a probability weighted average of a set of regrets occurring for each set of suitable pipelines; 12. The computing system of claim 11.

13. The program instructions cause the processor to: determining a rating value for the performance of each pipeline in the set of preferred pipelines on the new dataset; and calculating regrets incurred for each pipeline in the set of suitable pipelines for the new dataset; selecting at least one further suitable pipeline in the predetermined manner based on the rating value for the performance and the representative data set corresponding to the at least one further suitable pipeline having the lowest regret with respect to the new data set; processing said new data set in each of said at least one further suitable pipeline; 13. The computing system of claim 12, further comprising:

14. The program instructions cause the processor to: determining a rating value for the performance of each of the at least one further preferred pipeline with respect to the new data set; and responsive to said determining a rating value for the performance of each of said at least one further suitable pipeline with respect to said new data set, identifying among said at least one further pipeline a pipeline having a highest rating value with respect to said new data set; 14. The computing system of claim 13, further comprising:

15. 15. The computing system of claim 14, wherein the set of preferred pipelines comprises a set of pipelines that have been given the highest ratings on the corresponding representative data sets.

16. On your device, storing a plurality of rating values ​​for a plurality of machine learning pipelines and a plurality of N datasets previously processed by the plurality of machine learning pipelines, each rating value corresponding to performance of a pipeline in the plurality of machine learning pipelines and for a dataset in the plurality of N datasets; For each pair u in the plurality of N previously processed data sets, i and j, i=1 to N, j=1 to N, i≠j, ｉ , u ｊ On the other hand, u in the latent space ｉ From ｊ Distance d to ｉ，ｊ determining the distance d ｉ，ｊ Let, be the dataset u ｊ The pipeline selected in a predetermined manner based on a set of rating values ​​for the data set u ｉ , and the regret for the selected pipeline corresponds to a predicted value of regret that occurs when a rating is given to the performance of the selected pipeline with respect to the dataset u ｉ the determining step comprising a monotonically decreasing function of the rating value with respect to the performance of the pipeline with respect to clustering the plurality of N previously processed data sets in the latent space and identifying in each cluster a representative data set for which each distance to the data set from other data sets in the cluster is minimized across the cluster; in response to receiving a new dataset, selecting a set of preferred pipelines from the plurality of machine learning pipelines for processing the new dataset, each preferred pipeline being selected according to a set of rating values ​​for the representative dataset; processing said new data set in said set of suitable pipelines; A computer program for selecting a suitable machine learning pipeline for processing a new dataset, the computer program causing a machine learning pipeline to execute

17. The data set u ｊ The predetermined manner for selecting the pipeline based on the rating value for the dataset u ｊ as a random variable from a probability distribution conditioned on the rating value for the predicted value of the regret comprises a probability weighted average of a set of regrets occurring for each set of suitable pipelines; 17. A computer program product according to claim 16.

18. The device further comprises: determining a rating value for the performance of each pipeline in the set of suitable pipelines on the new dataset; calculating regrets incurred for each pipeline in the set of suitable pipelines for the new dataset; selecting at least one further suitable pipeline in the predetermined manner based on the rating value for the performance and the representative data set corresponding to the at least one further suitable pipeline having the lowest regret with respect to the new data set; processing said new data set in each of said at least one further suitable pipeline; 20. The computer program product of claim 17, further comprising:

19. The device further comprises: determining a rating value for the performance of each of the at least one further preferred pipeline with respect to the new data set; responsive to said determining a rating value for the performance of each of said at least one further suitable pipeline with respect to said new data set, identifying among said at least one further pipeline a pipeline having a highest rating value with respect to said new data set; 20. The computer program product of claim 18, further comprising:

20. 20. The computer program product of claim 19, wherein the set of preferred pipelines comprises a set of pipelines that are given the highest ratings on the corresponding representative dataset.

Images