Systems and methods for data classification

Abstract

This disclosure describes a method and system (120) for data classification. The system (120) includes at least one processor (230) coupled to the memory (240) and configured to receive at least one first dataset including at least one labeled dataset and at least one unlabeled dataset, and process the received labeled dataset to generate at least one first meta-feature including a cluster index. The processor (230) is further configured to estimate a classification performance score of each of a plurality of classification models for the at least one labeled dataset by associating the generated meta-features with pre-built models. The processor (230) is further configured to generate a list including the classification models sorted in descending order of the estimated performance scores, and select the top N classification models from the list to build an ensemble classification model for classifying the at least one unlabeled dataset.

Description

[Technical field]

[0001] The present disclosure relates generally to the field of automated machine learning. In particular, the present disclosure relates to systems and methods for data classification. [Background technology]

[0002] Machine learning is an important and rapidly growing field in computer science. It helps address a variety of real-world problems. Machine learning uses various concepts from statistics to build models that can learn patterns from historical data to predict new output values. Machine learning has potential growth in both academia and industry as it is applied across a wide variety of fields.

[0003] In the field of supervised machine learning, classification refers to a predictive modeling problem in which a class label is predicted for given data. Classification draws some conclusions from the input data given for training and predicts a class label / category for the given data. Classifying given data is a very important task in machine learning, for example, whether an email is spam or non-spam email, whether a transaction is fraudulent or not, etc. Since the applications of classification are vast, it becomes necessary to select the best classification model for a given dataset.

[0004] The performance of any machine learning classification task depends on the selection of the learning model, the selection of the classification model, and the characteristics of the dataset. Various classification models / methods have been introduced for data classification. Classification model selection is the process of identifying the appropriate classification model that is best suited to classify a given dataset. Selection of an appropriate classification model that maximizes performance for a given task is an essential step in data science. The traditional approach to select the best classification model is to train different classification models, evaluate their performance on a validation set, and select the best classification model. However, this approach is time-consuming, resource-intensive, and requires user intervention to select the best classification model.

[0005] Nowadays, various techniques have been introduced for automatic classification model selection, such as meta-learning, deep reinforcement learning, Bayesian optimization, evolutionary algorithms, and budget-based evaluation. These techniques automatically select a classification model for a given dataset. However, these automatic classification model techniques are also time-consuming and resource-intensive. Furthermore, with recent technological advances, the amount of data generated continues to increase. However, with conventional techniques, it is difficult to accurately classify huge datasets in real time.

[0006] Therefore, due to the vast and rapidly growing amount of data that needs to be classified, there is a need for further improvements in the technology, in particular time- and resource-efficient techniques that can automatically select the best classification model for a given dataset and can accurately classify a given dataset in real time, even when the dataset contains a vast amount of data.

[0007] Conventionally, there is no commercially available technology that can address the above problems. Therefore, there is a need for a technique that facilitates time- and resource-efficient automatic classification model selection for accurately classifying a given data set.

[0008] The information disclosed in the Background section is merely intended to enhance understanding of the general background of the present invention and should not be construed as an admission or any form of suggestion that this information forms prior art already known to those skilled in the art. Summary of the Invention

[0009] The present disclosure overcomes one or more of the shortcomings discussed above and provides additional advantages. Additional features and advantages are realized through the techniques of the present disclosure. Other embodiments and aspects of the present disclosure are described in detail herein and are considered a part of the disclosure.

[0010] The objective of this disclosure is to automatically recommend / select one or more best classification models.

[0011] Another objective of the present disclosure is to classify a given data set using one or more best classification models.

[0012] Another objective of the present disclosure is to accurately assign class labels to an unlabeled dataset in a time- and resource-efficient manner.

[0013] Another object of the present disclosure is to determine the classification complexity of a given data set.

[0014] Yet another object of the present disclosure is to provide a machine learning as a service platform for classification model building and data classification.

[0015] The above objects, as well as other objects, features, and advantages of the present disclosure will become apparent to those skilled in the art from a review of the following description, the accompanying drawings, and the appended claims.

[0016] According to one aspect of the present disclosure, a method and system for data classification is provided.

[0017] In a non-limiting embodiment of the disclosure, the present application discloses a method for data classification. The method may include receiving at least one first dataset, the at least one first dataset may include at least one labeled dataset and at least one unlabeled dataset. The method may further include processing the at least one labeled dataset to generate at least one first meta-feature from the at least one labeled dataset, the at least one first meta-feature being at least one first cluster index. The method may associate the at least one first meta-feature with a pre-constructed model including a plurality of classification models, the pre-constructed model may further include at least one mapping function for mapping the at least one pre-computed meta-feature to a plurality of pre-computed classification performance scores corresponding to the plurality of classification models. The method may further include estimating a classification performance score of each of the plurality of classification models for the at least one labeled dataset based on associating the at least one first meta-feature with the pre-constructed model. The method may further include generating a list including the plurality of classification models sorted in descending order of the estimated classification performance score, and selecting a predetermined number of the top classification models from the list to construct an ensemble classification model for classifying the at least one unlabeled dataset.

[0018] In another non-limiting embodiment of the present disclosure, classifying the at least one unlabeled dataset may further include processing the at least one unlabeled dataset using an ensemble classification model to predict a class label based on one of a majority vote, a weighted average, and model stacking.

[0019] In another non-limiting embodiment of the present disclosure, processing the at least one labeled dataset to generate at least one first meta-feature may include processing the at least one labeled dataset to generate at least one cleaned dataset; processing the at least one cleaned dataset using at least one clustering model to generate one or more clusters; and generating a multidimensional vector by processing the one or more clusters, where the multidimensional vector comprises the at least one first meta-feature.

[0020] In another non-limiting embodiment of the present disclosure, the method may further include determining a classification complexity of the at least one first data set by comparing the estimated classification performance score to a preset threshold.

[0021] In another non-limiting embodiment of the present disclosure, the pre-constructed model may be generated by: receiving at least one second dataset; processing the at least one second dataset to generate at least one training sub-dataset; processing the at least one training sub-dataset using at least one clustering model to generate one or more clusters; generating a multidimensional vector by processing the one or more clusters, wherein the multidimensional vector includes at least one second meta-feature corresponding to the at least one training sub-dataset, the at least one second meta-feature being at least one second cluster index; generating a plurality of classification performance scores corresponding to a plurality of classification models by processing the at least one training sub-dataset; and generating the pre-constructed model by associating the generated at least one second meta-feature with the generated plurality of classification performance scores, wherein the at least one second meta-feature corresponds to the at least one pre-computed meta-feature and the plurality of classification performance scores correspond to the plurality of pre-computed classification performance scores.

[0022] In another non-limiting embodiment of the present disclosure, generating a plurality of classification performance scores corresponding to the plurality of classification models may include generating a best classification performance score for each of the plurality of classification models by tuning one or more hyperparameters corresponding to the plurality of classification models.

[0023] In another non-limiting embodiment of the present disclosure, the present application discloses a system for data classification. The system may comprise a memory and at least one processor communicatively coupled to the memory. The at least one processor may be configured to receive at least one first dataset, the at least one first dataset including at least one labeled dataset and at least one unlabeled dataset. The at least one processor may be further configured to process the at least one labeled dataset to generate at least one first meta-feature from the at least one labeled dataset, the at least one first meta-feature being at least one first cluster index. The at least one processor may be further configured to associate the at least one first meta-feature with a pre-constructed model including a plurality of classification models. The pre-constructed model may further include at least one mapping function for mapping the at least one pre-computed meta-feature to a plurality of pre-computed classification performance scores corresponding to the plurality of classification models. The at least one processor may be further configured to estimate a classification performance score of each of the plurality of classification models for the at least one labeled dataset based on associating the at least one first meta-feature with the pre-constructed models, and generate a list including the plurality of classification models sorted in descending order of the estimated classification performance scores. The at least one processor may be further configured to select a predetermined number of the top classification models from the list to construct an ensemble classification model for classifying the at least one unlabeled dataset.

[0024] In another non-limiting embodiment of the present disclosure, the at least one processor may be configured to classify the at least one unlabeled dataset by processing the at least one unlabeled dataset using an ensemble classification model to predict a class label based on one of a majority vote, a weighted average, and model stacking.

[0025] In another non-limiting embodiment of the present disclosure, the at least one processor may include: processing the at least one labeled dataset to generate at least one cleaned dataset; processing the at least one labeled dataset to generate at least one first meta-feature; processing the at least one cleaned dataset using the at least one clustering model to generate one or more clusters; and processing the one or more clusters to generate a multidimensional vector, where the multidimensional vector includes the at least one first meta-feature.

[0026] In another non-limiting embodiment of the present disclosure, the at least one processor may be further configured to determine a classification complexity of the at least one first data set by comparing the estimated classification performance score to a preset threshold.

[0027] In another non-limiting embodiment of the present disclosure, the at least one processor may be further configured to generate the pre-constructed model by receiving at least one second dataset, processing the at least one second dataset to generate at least one training sub-dataset, and processing the at least one training sub-dataset using the at least one clustering model to generate one or more clusters. The at least one processor may be further configured to generate a multi-dimensional vector by processing the one or more clusters, the multi-dimensional vector including at least one second meta-feature corresponding to the at least one training sub-dataset, the at least one second meta-feature being at least one second cluster index. The at least one processor may be further configured to generate a plurality of classification performance scores corresponding to the plurality of classification models by processing the at least one training sub-dataset. The at least one processor may be further configured to generate a pre-constructed model by associating the generated at least one second meta-feature with the generated plurality of classification performance scores, where the at least one second meta-feature corresponds to the at least one pre-computed meta-feature and the plurality of classification performance scores correspond to the plurality of pre-computed classification performance scores.

[0028] In another non-limiting embodiment of the present disclosure, the at least one processor may be configured to generate multiple classification performance scores corresponding to the multiple classification models, and generate a best classification performance score for each of the multiple classification models by tuning one or more hyperparameters corresponding to the multiple classification models.

[0029] In another non-limiting embodiment of the present disclosure, the system may be configured to provide a machine learning as a service (MLaaS) platform for data classification and classification model selection.

[0030] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the exemplary aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. [Brief description of the drawings]

[0031] Further aspects and advantages of the present disclosure will be readily understood from the following detailed description when taken in conjunction with the accompanying drawings, in which: FIG. 1 is a block diagram of a method for manufacturing a semiconductor device according to the present disclosure; FIG. 2 is a block diagram of a semiconductor device according to the present disclosure;

[0032] [Figure 1] 1 illustrates an example environment of a communication system 100 for data classification, according to some embodiments of the present disclosure. [Diagram 2] 2 illustrates a block diagram 200 of the communication system 100 shown in FIG. 1, in accordance with some embodiments of the present disclosure. [Diagram 3] 3 illustrates a process flow diagram 300 for model selection and data classification, according to some embodiments of the present disclosure. [Figure 4] 4 shows a block diagram 400 of a computing system 110, 120 according to some embodiments of the present disclosure. [Diagram 5] 5 shows a flowchart 500 illustrating a method for data classification according to some embodiments of the present disclosure. [Figure 6] 6 shows a flowchart 600 illustrating a method for generating a trained / pre-built model according to some embodiments of the present disclosure.

[0033] Those skilled in the art should understand that any block diagrams herein represent conceptual views of example systems embodying the principles of the present disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like may be substantially represented in a computer-readable medium and thus represent various processes that may be performed by a computer or processor, whether or not such a computer or processor is explicitly illustrated. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present disclosure described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0035] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are described in detail below. It is to be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure.

[0036] The terms "comprise(s)", "comprising", "include(s)", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a setup, device, apparatus, system, or method that includes a list of components or steps does not include only those components or steps, but may include other components or steps that are not expressly listed or that are inherent to such setup or device or apparatus or system or method. In other words, one or more elements in a system followed by "comprises...a" does not, without further constraints, exclude the presence of other or additional elements in the device or system or apparatus.

[0037] Terms such as "at least one" and "one or more" may be used interchangeably throughout the description. Terms such as "a plurality of" and "multiple" may be used interchangeably throughout the description. Furthermore, terms such as "mapping function", "regressor", and "regression function" may be used interchangeably throughout the description. Furthermore, terms such as "pre-built model" and "trained model" may be used interchangeably throughout the description.

[0038] In the following detailed description of the embodiments of the present disclosure, reference is made to the accompanying drawings, which form a part of this specification and are shown as illustrations of specific embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure, and it should be understood that other embodiments may be utilized and changes may be made without departing from the scope of the present disclosure. Therefore, the following description should not be interpreted in a limiting sense. In the following description, well-known functions or configurations are not described in detail, as they would obscure the description with unnecessary detail.

[0039] In general, clustering is an unsupervised machine learning task, and classification is a supervised machine learning task. In this disclosure, a clustering index represents a cluster evaluation metric used to evaluate the quality of clusters induced by a clustering model for a given dataset. A clustering model groups datasets with similar characteristics into neighborhoods or isolates of different sizes. A clustering index measures the ability of a clustering model to induce good quality neighborhoods that share similar data characteristics. Thus, a clustering index represents a dataset characteristic with respect to a clustering model. In this disclosure, a clustering index is used as a meta-feature for classification model selection and for accurately classifying a given dataset.

[0040] In this disclosure, the term model suitability refers to the ability of a classification model to learn a classification task for a given dataset. The actual model suitability for a dataset can be measured based on the expected classification performance of a classification model for a given dataset. The F1 score is used as the classification performance metric in this disclosure.

[0041] In this disclosure, the term classification complexity refers to the difficulty of learning a classification model for a given dataset.

[0042] In machine learning, the classification task is a discriminant function that maps the characteristics of a dataset to an appropriate output category. In general, a discriminant function is a function of several variables that is used to assign items to one of two or more groups. Machine learning classification models are defined by their ability to generalize and classify unobserved data.

[0043] This disclosure provides techniques (methods and systems) for data classification and model selection. As described in the Background section, conventional techniques for classification model selection are time-consuming and resource-intensive, and it is difficult to perform accurate classification of large datasets in real time using conventional techniques.

[0044] To overcome these and other problems, the present disclosure proposes a technique that uses a clustering index to automatically select one or more classification models from multiple available classification models to form an ensemble classification model. The ensemble classification model can be used to accurately classify a given dataset. The present disclosure builds an ensemble classification model without fitting / training multiple classification models across a dataset, using a clustering index as a data characteristic (or meta-feature) to select the best classification model. The present disclosure can provide users with a machine learning as a service (MLaaS) platform for data classification and classification model selection.

[0045] In recent years, the demand for machine learning as a service has increased with the increase in data sources. Companies across industries are leveraging the power of machine learning at various stages of their product cycle. This has paved the way for companies to offer machine learning as a service. A functional and ready-to-use machine learning as a service (MLaaS) platform can be beneficial for small businesses, developers, and researchers to build their own solutions. This helps to overcome the need for high computational resources and time spent. The proposed system of the present disclosure can be utilized as a service for machine learning model building. In particular, the ensemble classification model can be provided to users / clients either as a predictive application programming interface (API) or as a deployable solution.

[0046] Reference is made to FIG. 1, which illustrates a communication system 100 for use in data classification and model selection, according to some embodiments of the present disclosure. The communication system 100 may comprise a first computing system 110 (or a client computing system) that may communicate with one or more first data sources 130. The one or more first data sources 130 may include at least one first data set 160 on which classification is performed. The communication system 100 may further comprise a second computing system 120 (or a server) that communicates with the first computing system 110 via at least one network 150. Furthermore, the second computing system 120 may communicate with one or more second data sources 140. The one or more second data sources 140 may include at least one second data set 160 for training the second computing system 120.

[0047] Network 150 may include data networks such as the Internet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), etc. In particular embodiments, network 150 may include a wireless network, such as, but not limited to, a cellular network, and may use a variety of technologies including Enhanced Data rates for Global Evolution (EDGE), General Packet Radio Service (GPRS), Global System for Mobile Communications (GSM), Internet protocol Multimedia Subsystem (IMS), Universal Mobile Telecommunications System (UMTS), etc. In one embodiment, network 150 may include or cover a network or sub-network, each of which may include, for example, a wired or wireless data path.

[0048] The first and second data sources 130, 140 may be any data source containing a large amount of data and / or information. The first and second data sources 130, 140 may be any public or private data source, including but not limited to bank records, IoT logs, computerized medical records, online shopping records, user chat data stored on a server, computing device logs, vulnerability databases, etc. The first computing system 110 may fetch / receive at least one first data set 160 from at least one first data source 140, and the second computing system 110 may fetch / receive at least one second data set 170 from at least one second data source 130.

[0049] FIG. 1 will now be described in conjunction with FIG. 2, which is a block diagram 200 of a communication system 100, according to some embodiments of the present disclosure. According to one embodiment of the present disclosure, the communication system 100, 200 may comprise a first computing system 110, a second computing system 120, at least one first source 130, and at least one second source 140. The first computing system 110 may comprise at least one first processor 210 and at least one first memory 220. Similarly, the second computing system 120 may comprise at least one second processor 230 and at least one second memory 240.

[0050] The first and second processors 210, 230 may include, but are not limited to, a general purpose processor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a microprocessor, a microcomputer, a microcontroller, a central processing unit, a state machine, a logic circuit, and / or any device that manipulates signals based on operational instructions. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration.

[0051] The first memory 220 may be communicatively coupled to at least one first processor 210, and the second memory 240 may be communicatively coupled to at least one second processor 230. The first and second memories 220, 240 may include various instructions, one or more datasets, and one or more clusters, one or more class labels, one or more classification models, one or more clustering models, etc. The first and second memories 220, 240 may include a random access memory (RAM) unit and / or a non-volatile memory unit, such as a read-only memory (ROM), an optical disk drive, a magnetic disk drive, a flash memory, an electrically erasable read-only memory (EEPROM), memory space on a server or a cloud, etc.

[0052] The communication system 100 proposed in this disclosure can be referred to as a data classification system that can build a trained model, select at least one classification model using the trained model, form an ensemble classification model using the selected at least one classification model, and classify a given data set using the ensemble classification model.

[0053] In one non-limiting embodiment of the present disclosure, the at least one first processor 210 can extract at least one first data set 160 from the at least one first data source 130. In one non-limiting embodiment, the one or more data sets 160 may be transmitted to the first processor 210. The at least one first processor 210 can transmit the at least one first data set 160 to the second at least one second processor 230 of the second computing system 120. The at least one second processor 230 can process the received at least one first data set 160 to assign one or more class labels. The at least one second processor 230 uses a pre-built / trained model for data classification. The processing in the at least one second processor 230 is described below with the aid of a process flow diagram 300 as illustrated in FIG. 3.

[0054] The second computing system 120 may operate in two phases: a first phase, which is a training phase 302, and a second phase, which is a prediction phase 304. It may be noted here that the second computing system 120 is trained first, and model selection and data classification are performed thereafter. The result of the training phase 302 is a trained model or pre-built model 320. The terms "trained model" and "pre-built model" are used interchangeably throughout the description.

[0055] The training phase 302 may be further divided into three sub-phases: a pre-processing phase 306, a dataset building phase 308, and a mapper phase 310. The prediction phase 304 may be further divided into two sub-phases: a recommendation phase 312 and a model building / classification phase 314. The recommendation phase 312 may include some or all of the functionality of the pre-processing phase 306 and the dataset building phase 308 of the training phase 302. The different phases are described in detail below. Training Phase:

[0056] In one non-limiting embodiment of the present disclosure, the at least one second processor 230 may receive or fetch at least one second data set 170 from at least one second data source 140. The at least one second data set 170 may be collectively referred to as D T and can contain one or more datasets: D T ={D1,D2,D3,...,D n} (1)

[0057] In one non-limiting embodiment of the present disclosure, the pre-processing phase 306 divides at least one second data set 170 into a set B of several sub-data sets (or sub-samples) generated by stratified random sampling with replacement. TIn one subtask, the at least one second processor 230 may perform a cleaning operation on the received at least one second dataset 170 to generate at least one cleaned dataset. Data cleaning identifies and removes errors and duplicate data from the at least one second dataset 170 to create a reliable dataset. Data cleaning improves the quality of the training data and enables accurate decision-making. Cleaning the at least one second dataset 170 may include, but is not limited to, normalizing the at least one second dataset 170, dropping empty cells from the at least one second dataset 170, standardizing the at least one second dataset 170, and the like. The purpose of cleaning is to remove unnecessary data from the at least one second dataset 170 to make the dataset uniform and understandable to various machine learning models. Cleaning the at least one second dataset 170 at an early stage can reduce unnecessary calculations in subsequent phases, thereby saving the overall time of the training phase 302.

[0058] In one non-limiting embodiment of the present disclosure, the at least one second processor 230 may split the cleaned dataset into a predetermined ratio of a training dataset and a test dataset. In one non-limiting embodiment, the predetermined ratio may be 70:30 or 80:20. The training dataset may be used to train the computing system 120 to generate the trained model 320, and the test dataset may be used to cross-validate the trained model 320. The test dataset may be referred to as a validation dataset.

[0059] In one non-limiting embodiment of the present disclosure, the training data set and the test data set may be subjected to independent sampling to generate the respective sub-data sets, i.e., at least one training sub-data set may be generated from the training data set, and at least one test sub-data set may be generated from the test data set. The sampling used here is stratified random sampling with replacement. Note that sampling (i.e., construction of multiple sub-data sets) results in an increase in the number of data sets for training the pre-constructed model 320, and the more training data sets there are, the better the model generated and the higher the accuracy. Another advantage of using sub-data sets is that it provides a wider coverage of the data set variance characteristic of the regression function. The set of training sub-data sets is a set of sub-data sets, B T It can be expressed as: B T ={B1,B2,B3,...,B n} (2) The test subdataset is set B T The output of the pre-processing phase 306 is a sub-dataset that is provided as input to the dataset construction phase 308.

[0060] In one non-limiting embodiment of the present disclosure, at least one second processor 230 in the dataset construction phase 308 may receive the generated training and test sub-datasets and process them to generate one or more multi-dimensional vectors. The processing in the dataset construction phase 308 occurs in two parallel steps 316 and 318. In one non-limiting embodiment of the present disclosure, at least one clustering model and multiple classification models may be predefined / pre-fed to the at least one second processor 230. The at least one clustering model may be collectively represented as A and the multiple classification models may be collectively represented as C. A = {A1,A2,A3,...,A n} (3) C={C1,C2,C3,...,C n} (4)

[0061] In a first step 316 of the dataset construction phase 308, the at least one second processor 230 may process the at least one training sub-dataset using at least one clustering model A to generate at least one cluster for each clustering model. The clusters generated using the at least one clustering model may be collectively represented as a multi-dimensional vector CL, which may include different clusters generated by different clustering models. CL={CL1,CL2,CL3,...,CL n} (5) Here, CL i is the clustering model C i Each of the sets of clusters may further include at least one cluster as follows: Clusters generated by model A1: CL1 = {CL 11 ,CL 12 ,CL 13 ,...,CL 1n} Clusters generated by model A2: CL2 = {CL 21 ,CL 22 ,CL 23 ,...,CL 2n} Clusters generated by model A3: CL3 = {CL 31 ,CL 32 ,CL 33 ,...,CL 3n} Clusters generated by model Am:CL m ={CL m1 ,CL m2 ,CL m3 ,...,CL mn}

[0062] After generating at least one cluster for each clustering model, the at least one second processor 230 may process each of the generated clusters to extract meta-features from each of the generated clusters. The meta-features, also referred to as data characteristics, can characterize the complexity of the dataset and provide estimates of the performance of different clustering models. In the present disclosure, a clustering index is a feature of at least one second dataset D. T It is used as a meta-feature that represents different characteristics of a given dataset. Here, it can be noted that the clustering index has a strong correlation with the performance of a classification / clustering model for a given dataset. Different clustering models have different clustering assumptions for grouping sub-datasets into neighborhoods. When clustering indexes measure the performance of such clustering algorithms, they essentially capture different characteristics of the sub-datasets. In general, the clustering index is a measure for validating the clusters induced by the clustering model.

[0063] Clustering indexes can be divided into two categories: internal clustering indexes and external clustering indexes. If a clustering index does not rely on external information such as data labels, the index is called an internal clustering index or a quality index. Conversely, if a clustering index uses data point labels, the index is called an external clustering index. Thus, external clustering indexes require a priori data to evaluate the results of the clustering model, while internal clustering indexes do not. Some of the most commonly used clustering indexes are: Internal clustering indices: Variance, Banfeld-Raftery, Ball-Hall, PBM, Det ratio, Log-Det ratio, Ksq-DetW, Score, Silhouette, Log-SS ratio, C-index, Dunn, Ray-Turi, Calinski-Harabasz, Trace-WiB, Davies-Bouldin, etc. External clustering indices: Entropy, Purity, Recall, Folkes-Mallows, Rogers-Tanimoto, F1, Kulczynski, Norm-Mutual Information, Sokal-Sneath, Rand, Hubert, Homogeneity, Completeness, V-Measure, Jaccard, Adj-Rand, Phi, McNemar, Russell-Rao, Precision and more.

[0064] At least one desired clustering index may be preselected and provided to the at least one second processor 230. The at least one second processor 230 may then determine a value of the at least one desired clustering index for the generated clusters of each clustering model. The value of the clustering index may be determined using conventional known techniques. The at least one second processor 230 may then determine a final clustering index for the particular clustering model by averaging corresponding clustering indexes of different clusters of the particular clustering model to generate a multi-dimensional vector of clustering indexes. The multi-dimensional vector of clustering indexes may be calculated using the following formula: T where the multidimensional vector I T The generation of is explained as an example.

[0065] Consider an example in which two clustering algorithms A1 and A2 are used to cluster a sub-dataset, and there are two clusters generated by each of the clustering models A1, A2. Clustering model A={A1,A2} Clusters in the first clustering model A1: CL1 = {CL 11 ,CL 12} Clusters in the second clustering model A2: CL2 = {CL 21 ,CL 22} Consider the case where two clustering indexes I1 and I2 are used as meta-features. The at least one second processor 230 can determine the values ​​of I1 and I2 for each of the generated clusters. CL 11 The value of the first clustering index I1 for 111 CL 12 The value of the first clustering index I1 for 112 CL 11 The value of the second clustering index I2 for 211 CL 12 The value of the second clustering index I2 for 212

[0066] Then, the at least one second processor 230 selects different clusters CL of the first clustering model A1. 11 , C.L. 12 The value of the first clustering index I1 generated for I 111 , I 112 A value of the first clustering index I1 for the first clustering model A1 can be determined by taking the average of That is, the value of the first clustering index I1 for the first clustering model A1: I 11 =avg(I 111 ,I 112 ) Similarly, Value of the second clustering index I2 for the first clustering model A1: I 21 =avg(I 211 ,I 212 ) Here, the values ​​of the clustering indexes I1 and I2 for the first clustering model A1 have been determined. Similarly, the at least one second processor 230 determines the values ​​of the clustering indexes I1 and I2 for the second clustering model A2 (i.e., I 12 and I 22 ) can then be determined. The values ​​of the clustering indices of the two clustering models A1 and A2 are then concatenated to obtain a multidimensional vector of clustering indices I T can be formed.

number

[0067] Similarly, the values ​​of the clustering indices for all of the at least one clustering model are determined, and a vector I T The cations may be linked in the following manner:

number

[0068] In a second step 318 of the dataset construction phase 308, the at least one second processor 230 generates a plurality of classification models C={C1, C2, C3, ..., C n} may be generated. The classification performance score of the classification model for the dataset may indicate the maximum achievable classification performance of the classification model, measured as a model fit score. The classification performance may be measured using the F1 score, which is a weighted average of precision and recall. The value of the F1 score may be between 0 and 1, with 1 being the best score and 0 being the worst score. The classification performance of the different classification models may be calculated using the vector O T These can be collectively represented as: O T ={O1,O2,O3,...,O n} (7) In one non-limiting embodiment of the present disclosure, the at least one second processor 230 may generate a best classification performance score for each of the multiple classification models by adjusting one or more hyperparameters corresponding to the multiple classification models. In one non-limiting embodiment, each classification model may have its own hyperparameters. For example, the classification model "logistic regression" may have penalty and tolerance as its hyperparameters. Some exemplary classification models and their hyperparameters are listed in Table 1 below. [Table 1]

[0069] Here, the vector O T Let us explain the generation of the training sub-dataset B by using an example. T In the set of, there are two classification models, C1 and C2, and three training sub-datasets, B1, B2, and B3. Classification model C={C1,C2} Training subdataset B T ={B1,B2,B3} O ij But subdataset B j Classification model C for i represents the classification performance score of Classification performance score of C1 on subdataset B1 = O 11 Classification performance score of C1 on subdataset B2 = O 12 Classification performance score of C1 on subdataset B3 = O 13 Classification performance score of C2 on subdataset B1 = O 21 Classification performance score of C2 on subdataset B2 = O 22 Classification performance score of C2 on subdataset B3 = O 23

[0070] In one non-limiting embodiment, Data Set B T The classification performance score of the classification model C1 for the whole dataset may be denoted as O1. T The classification performance score of the classification model C1 for the whole may be represented as O2, where, to determine the classification performance score O1, the at least one second processor 230 calculates the classification performance score O 11 , O 12、 O 13 can be averaged. That is, O1=avg{O 11 ,O 12, O 13} Similarly, O2=avg(O 21 ,O 22, O 23 ). Here, the multidimensional vector O of the classification models C1 and C2 T can be expressed as: O T ={O1,O2}. A multidimensional vector O of multiple classification models C T can be expressed as follows: O T ={O1,O2,O3,...,O n}. The output of the second step 318 is a multi-dimensional vector O T It is.

[0071] In one non-limiting embodiment of the present disclosure, the mapper phase 310 receives two different vectors / datasets from the dataset construction phase 308, namely, one vector of cluster indexes I T and another vector of classification performance scores O THere, it may be noted that there exists a strong correlation between the clustering index of a dataset under a particular clustering assumption and its maximum achievable classification performance score measured in terms of F1 score for different classification models. This correlation may be modeled as one or more regression functions (or regressors) for multiple classification models. In general, regression is a machine learning technique that helps predict a continuous outcome variable (y) based on the values ​​of one or more predictor variables (x). Briefly, the goal of a regression function is to construct a mathematical equation that defines a variable (y) as a function of variables (x). The one or more regression functions may be collectively represented as R. R = {R1, R2, R3, ..., R n} (8) In this disclosure, regression functions are sometimes referred to as mapping functions. The goal of the mapper phase 310 is to build a model 320 that is trained using one or more mapping / regression functions.

[0072] In one non-limiting embodiment of the present disclosure, the at least one second processor 230 may generate a vector (I T ,O T ) can be used as training data to train one or more regression functions R. R:I T →O T (9) The at least one second processor 230 can evaluate the performance of the regression function using an R-squared (R2) metric. R-squared is a statistical measure that represents the proportion of variance in a dependent variable that is explained by the independent variable(s) in a regression function. One or more hyperparameters of the regressor function R can be tuned using cross-validation on the training sub-dataset. In this way, the regression function that gives the best performance for the at least one second data set can be selected. In one non-limiting embodiment, the at least one second processor 230 can build individual regression functions for the multiple classification models instead of a single regression function for all classification models. The best performing regression function constitutes the trained model or pre-built model 320. Once the trained model 320 is generated, the training phase 302 ends.

[0073] In a non-limiting embodiment of the present invention, when each dataset of the at least one second dataset 170 is considered as a single instance vector of the clustering index, the number of training samples is limited by the number of datasets present in the at least one second dataset 170. This makes it difficult to learn the regressor function due to the lack of training samples. Therefore, the regression function is trained using sub-datasets instead of the full dataset. In this process, all datasets undergo an expansion by random sampling with replacement to generate multiple training sub-datasets. The advantage of using sub-datasets instead of the full dataset is more variability in the datasets used to train the regression function, making the regression function robust to the variance of the datasets. Another advantage is that it is easier to generate a clustering index from the sub-datasets compared to dealing with a large dataset in a single shot.

[0074] Prediction phase: In one non-limiting embodiment of the present disclosure, the trained / pre-built model 320 may be utilized in the prediction phase 304 for predicting class labels or recommending a classification model for the at least one first dataset 160. In the recommendation phase 312, the at least one second processor 230 may receive the at least one first dataset 160. The at least one first dataset 160 may be collectively referred to as D P and may include one or more data sets. D P ={D1',D2',D3',...,D n '} (10) The at least one first dataset 160 may include at least one labeled dataset and at least one unlabeled dataset. The at least one labeled dataset may be used to build / train one or more classification models. The at least one second processor 230 may use the built classification models to classify the at least one unlabeled dataset.

[0075] In one non-limiting embodiment of the present disclosure, in block 322, the at least one second processor 230 may process the received at least one labeled dataset to generate at least one first meta-feature from the at least one labeled dataset. In the present disclosure, the meta-feature is a cluster index. First, the at least one second processor 230 may pre-process the received at least one labeled dataset to generate at least one cleaned dataset. Then, the at least one second processor 230 may generate one or more sub-datasets from the at least one cleaned dataset. The one or more sub-datasets may be expressed as follows: B P ={B1',B2',B3',...,B n '} (11)

[0076] In one non-limiting embodiment of the present disclosure, the at least one second processor 230 may process the at least one cleaned dataset using at least one clustering model to generate at least one first cluster. The at least one cluster generated in the training phase 302 may be referred to as at least one second cluster. The at least one second processor 230 may then process the at least one first cluster to generate a multidimensional vector comprising at least one first cluster index. It is noted here that a detailed description of data cleaning, sub-dataset generation, cluster index generation has already been described while describing the training phase 302. Therefore, the same is omitted here for the sake of brevity. The at least one first cluster index is a multidimensional vector I P and may comprise one or more cluster indices for each of the at least one clustering model similar to equation (6). Here, the objective of the recommendation phase 312 is to find a classification performance score for each of the multiple classification models on at least one labeled dataset.

[0077] In one non-limiting embodiment of the present disclosure, the at least one second processor 230 may use the at least one first cluster index to query the pre-built model 320. In particular, the at least one second processor 230 may associate the at least one first cluster index with the pre-built model 320 including a plurality of classification models. As described above, the pre-built model 320 may include at least one best mapping function R for mapping the at least one meta-feature to a plurality of classification performance scores corresponding to the plurality of classification models. The at least one second processor 230 may estimate / predict a classification performance score of each of the plurality of classification models for the at least one labeled dataset based on associating the at least one first cluster index with the pre-built model 320.

[0078] In one non-limiting embodiment of the present disclosure, a multidimensional vector I includes at least one first cluster index. P may be input to at least one mapping function R of the pre-constructed model 320 to estimate a predicted classification performance score or model suitability score for each of the multiple classification models C. The estimated classification performance score 324 for a particular classification model for at least one labeled dataset is calculated based on the sub-dataset B. P The estimated classification performance scores 324 may be obtained by averaging the estimated classification performance scores 324 of a particular classification model for each data set of O. The estimated classification performance scores 324 may be collectively P and can be calculated as follows: O P ←R(I P ) (12) and O P ={O1',O2',O3',...,O n '} (13) Thus, using the techniques described in this disclosure, classification performance scores for different classification models can be predicted without even training them over the at least one first dataset 160. The prediction is based on the clustering index extracted from the at least one first dataset 160.

[0079] In one non-limiting embodiment of the present disclosure, after estimating the classification performance score of each of the plurality of classification models for the at least one labeled dataset, the at least one second processor 230 may generate an ordered list of the plurality of classification models. The ordered list may include the plurality of classification models arranged in descending order of the estimated classification performance scores 324 (i.e., the classification model with the highest classification performance score is placed at the top of the list and the classification model with the lowest classification performance score is placed at the bottom of the list). Thus, using the techniques of the present disclosure, the best classification model may be recommended for the at least one first dataset based on the estimated classification performance scores 324.

[0080] In one non-limiting embodiment of the present disclosure, the at least one second processor 230 may select a predetermined number (N) of top classification models from the ordered list to build an ensemble classification model 326. In the model building / classification phase 314, the at least one second processor 230 may use the at least one labeled dataset to select the TOP models with the best parameter settings. N Only this classification model can be built / trained.

[0081] The at least one second processor 230 may then receive the at least one unlabeled dataset. The at least one second processor 230 may use the ensemble classification model 326 to classify the at least one unlabeled dataset or to predict a class label for the at least one unlabeled dataset. To predict the class label, the at least one second processor 230 may use the TOP N The class classification models can be used to generate predictions of class labels, and their outputs can be combined using any one of majority voting, weighted averaging, and model stacking to predict the class label of at least one unlabeled dataset.

[0082] Thus, the present disclosure describes a technique for using a clustering index as a meta-feature for automatically selecting and recommending one or more classification models from a plurality of classification models. The disclosed data classification and model selection techniques are time-efficient and require fewer computational resources. The disclosed techniques have higher accuracy compared to other techniques for data classification.

[0083] In one non-limiting embodiment of the present disclosure, hyperparameters control the behavior of the computing system 120. The hyperparameters can be tuned by trial and error. Examples of hyperparameters can be the number of clusters and the number of training sub-datasets. The number of clusters is an important parameter for most clustering models. The number of clusters can be set to a value that gives the best results. Similarly, the number of training sub-samples can be set to a value that gives the best results.

[0084] In one non-limiting embodiment of the present disclosure, the at least one second processor 230 can determine a classification complexity of the at least one first data set 160. The classification complexity can indicate the difficulty of learning a classification model for a given data set. The at least one second processor 230 can generate an estimated classification performance score O P may be compared to a predetermined threshold. If the value of any estimated classification performance score is less than the predetermined threshold, the classification complexity is higher and the at least one first data set 160 is difficult to learn. On the other hand, if all the values ​​of the estimated classification performance scores are equal to or greater than the predetermined threshold, the classification complexity is lower and the at least one first data set 160 is easy to learn. Note that the value of the predetermined threshold may be based on trial and error.

[0085] Therefore, the present disclosure can estimate the classification complexity of at least one first dataset for a model class before classification model selection, and it becomes relatively easy to choose an appropriate classification model to solve the classification problem. This is particularly useful when dealing with large datasets, because it is tedious and time-consuming to evaluate different classification models in a large population for classification model selection.

[0086] In one non-limiting embodiment of the present disclosure, the proposed automatic model classification technique can be extended to an automatic machine learning platform to provide classification modeling as a service. In particular, the techniques of the present disclosure can provide a machine learning as a service platform where clustering indexes are used as data characteristics for classification model selection and advanced machine learning models can be built. A functional, ready-to-use machine learning as a service (MLaaS) platform is beneficial for organizations, developers, and researchers to go through the learning curve of how this paradigm works and help them build their solutions. It saves them from the high cost of computational and human resources.

[0087] The MLaaS platform may be provided to users in the form of an application programming interface (API) or a deployable solution. A client may upload at least one first dataset, and the platform may provide the client with class labels or a recommendation model for classification. This saves additional computational costs and improves user experience.

[0088] Thus, the techniques of this disclosure can provide faster classification of data and provide more accurate class labels in real time (even for large datasets).

[0089] Reference is now made to FIG. 4, which illustrates a block diagram of a computing system 110, 120 according to some embodiments of the present disclosure. In one non-limiting embodiment of the present disclosure, the computing system 110, 120 may comprise various other hardware components, such as various interfaces 402, memory 408, and various units or means, as illustrated in FIG. 4. These units may comprise a receiving unit 414, a processing unit 416, a transmitting unit 418, an associating unit 420, an estimating unit 422, a generating unit 424, a selecting unit 426, a determining unit 428, and various other units 430. The other units 430 may include a display unit, an identifying unit, a mapping unit, and the like. In one embodiment, the units 414-430 may be dedicated hardware units capable of executing one or more instructions stored in the memory 408 to perform various operations of the computing system 110, 120. In another embodiment, the units 414 - 430 may be software modules stored in the memory 408 that may be executed by at least one processor 210 , 230 to perform the operations of the computing systems 110 , 120 .

[0090] The interface 402 may include various software and hardware interfaces, such as a web interface, a graphical user interface, an input device-output device (I / O) interface 406, a network interface 404, etc. The I / O interface 406 may enable the computing systems 110, 120 to interact with other computing systems directly or through other devices. The network interface 404 may enable the computing systems 110, 120 to interact with one or more data sources 130, 140 directly or through a network 150.

[0091] The memory 408 may include one or more datasets 410, as well as various other types of data 412 (such as one or more cleaned datasets, one or more cluster indexes, one or more clustering models, one or more classification models, one or more classification performance scores, one or more training and testing datasets, etc.). The memory 408 may further store one or more instructions executable by at least the processor 210, 230. The memory 408 may be any of the memories 240, 260.

[0092] 5, a flow chart is depicted illustrating an exemplary method 500 for data classification, according to one embodiment of the present disclosure. Method 500 is provided for illustrative purposes only, and embodiments are intended to include or otherwise cover any method or procedure for generating at least one pattern from at least one data set.

[0093] The method 500 may include receiving at least one first data set, at block 502. The at least one first data set may include at least one labeled data set and at least one unlabeled data set, and may be received by the at least one second processor 230 from the at least one first processor 210. The operations of block 502 may be performed by the at least one second processor 230 of FIG. 2 or by the receiving unit 414 of FIG. 4.

[0094] At block 504, the method 500 may include processing the at least one labeled dataset to generate at least one first meta-feature from the at least one labeled dataset. The at least one first meta-feature may be at least one first cluster index. For example, the at least one second processor 230 may be configured to process the at least one labeled dataset to generate at least one first meta-feature from the at least one labeled dataset. The operation of block 504 may be performed by the processing unit 416 of FIG. 4.

[0095] In one non-limiting embodiment of the present disclosure, the operation of block 504, i.e., processing the at least one labeled dataset to generate at least one first meta-feature, may include processing the at least one labeled dataset to generate at least one cleaned dataset, and processing the at least one cleaned dataset using at least one clustering model to generate one or more clusters. For example, the at least one second processor 230 of FIG. 2 or the processing unit 416 of FIG. 4 may be configured to process the at least one labeled dataset to generate at least one cleaned dataset, and process the at least one cleaned dataset using the at least one clustering model to generate one or more clusters.

[0096] In one non-limiting embodiment of the present disclosure, the operation of block 504, i.e., processing the at least one labeled dataset to generate at least one first meta-feature, may further include generating a multi-dimensional vector by processing one or more clusters. For example, the at least one second processor 230 of FIG. 2 or the generating unit 424 of FIG. 4 may be configured to generate a multi-dimensional vector by processing one or more clusters. The multi-dimensional vector may include the at least one first meta-feature.

[0097] At block 506, the method 500 may include associating the at least one first meta-feature with a pre-constructed model. For example, the at least one second processor 230 of FIG. 2 or the associating unit 420 of FIG. 4 may be configured to associate the at least one first meta-feature with the pre-constructed model. The pre-constructed model may include a plurality of classification models. The pre-constructed model may further include at least one mapping function for mapping the at least one pre-computed meta-feature to a plurality of pre-computed classification performance scores corresponding to the plurality of classification models.

[0098] At block 508, the method 500 may further include estimating a classification performance score of each of the plurality of classification models for the at least one labeled dataset based on associating the at least one first meta-feature with the pre-constructed model. For example, the at least one second processor 230 of FIG. 2 or the estimation unit 422 of FIG. 4 may be configured to estimate a classification performance score of each of the plurality of classification models for the at least one labeled dataset based on associating the at least one first meta-feature with the pre-constructed model.

[0099] At block 510, the method 500 may include generating a list including the plurality of classification models sorted in descending order of the estimated classification performance score. For example, the at least one second processor 230 of FIG. 2 or the generating unit 424 of FIG. 4 may be configured to generate a list including the plurality of classification models sorted in descending order of the estimated classification performance score.

[0100] At block 512, the method 500 may include selecting a predetermined number of the top classification models from the list to build an ensemble classification model for classifying the at least one unlabeled dataset. For example, the at least one second processor 230 of FIG. 2 or the selection unit 426 of FIG. 4 may be configured to select a predetermined number of the top classification models from the list to build an ensemble classification model for classifying the at least one unlabeled dataset.

[0101] In one non-limiting embodiment of the present disclosure, classifying the at least one unlabeled dataset may further include processing the at least one unlabeled dataset using an ensemble classification model to predict a class label based on one of a majority vote, a weighted average, and model stacking. For example, the at least one second processor 230 of FIG. 2 or the processing unit 416 of FIG. 4 may be configured to process the at least one unlabeled dataset using an ensemble classification model to predict a class label based on one of a majority vote, a weighted average, and model stacking.

[0102] At block 514, the method 500 may include determining a classification complexity of the at least one first data set by comparing the estimated classification performance score to a preset threshold. For example, the at least one second processor 230 of FIG. 2 or the determination unit 428 of FIG. 4 may be configured to determine a classification complexity of the at least one first data set by comparing the estimated classification performance score to a preset threshold.

[0103] 6, a flow chart is depicted illustrating an exemplary method 600 for generating a pre-built model 320, according to one embodiment of the present disclosure. Method 600 is provided for illustrative purposes only, and embodiments are intended to include or otherwise cover any method or procedure for generating at least one pattern from at least one dataset.

[0104] The method 600 may include receiving or extracting at least one second data set, at block 602. The at least one second data set may be received by the at least one second processor 230 from the at least one first processor 210. The operations of block 602 may be performed by the at least one second processor 230 of FIG. 2 or by the receiving unit 414 of FIG. 4.

[0105] At block 604, the method 600 may include processing the at least one second data set to generate at least one training sub-data set. For example, the at least one second processor 230 of FIG. 2 or the processing unit 416 of FIG. 4 may be configured to process the at least one second data set to generate the at least one training sub-data set.

[0106] At block 606, the method 600 may include processing the at least one training sub-data set using the at least one clustering model to generate one or more clusters. For example, the at least one second processor 230 of FIG. 2 or the processing unit 416 of FIG. 4 may be configured to process the at least one training sub-data set using the at least one clustering model to generate one or more clusters.

[0107] At block 608, the method 600 may include generating a multidimensional vector by processing one or more clusters. For example, the at least one second processor 230 of FIG. 2 or the generating unit 424 of FIG. 4 may be configured to generate a multidimensional vector by processing one or more clusters. The multidimensional vector includes at least one second meta-feature corresponding to the at least one training sub-dataset. The at least one second meta-feature may be at least one second cluster index.

[0108] At block 610, the method 600 may include generating a plurality of classification performance scores corresponding to the plurality of classification models by processing the at least one training sub-data set. For example, the at least one second processor 230 of FIG. 2 or the generation unit 424 of FIG. 4 may be configured to generate a plurality of classification performance scores corresponding to the plurality of classification models by processing the at least one training sub-data set.

[0109] In a non-limiting embodiment of the present disclosure, the operation of block 610, i.e., generating a plurality of classification performance scores corresponding to the plurality of classification models, may include generating a best classification performance score for each of the plurality of classification models by tuning one or more hyperparameters corresponding to the plurality of classification models. For example, the at least one second processor 230 of FIG. 2 or the generating unit 424 of FIG. 4 may be configured to generate a best classification performance score for each of the plurality of classification models by tuning one or more hyperparameters corresponding to the plurality of classification models.

[0110] At block 612, the method 600 may include generating a pre-constructed model by associating the generated at least one second meta-feature with the generated plurality of classification performance scores. For example, the at least one second processor 230 of FIG. 2 or the generating unit 424 of FIG. 4 may be configured to generate the pre-constructed model by associating the generated at least one second meta-feature with the generated plurality of classification performance scores. The at least one second meta-feature may correspond to the at least one pre-computed meta-feature, and the plurality of classification performance scores may correspond to the plurality of pre-computed classification performance scores.

[0111] The above methods 500, 600 can be described in the general context of computer-executable instructions. Generally, computer-executable instructions can include routines, programs, objects, configuration information, data structures, procedures, modules, and functions that perform particular functions or implement particular abstract data types.

[0112] The order in which various operations of the method are described is not intended to be construed as a limitation, and any number of the described method blocks may be combined in any order to implement the method. Additionally, individual blocks may be deleted from the method without departing from the spirit and scope of the subject matter described herein. Further, the method may be implemented in any suitable hardware, software, firmware, or combination thereof.

[0113] The various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions, which may include various hardware and / or software components and / or modules, including but not limited to the processors 210, 230 of Figure 2 and the various units of Figure 4. Generally, where there are operations illustrated in the figures, those operations may have corresponding counterpart means-plus-function components.

[0114] It should be noted here that the subject matter of some or all of the embodiments described with reference to Figures 1 to 4 may be relevant to the method and will not be repeated for the sake of brevity.

[0115] In non-limiting embodiments of the present disclosure, one or more non-transitory computer-readable media may be utilized to implement embodiments consistent with the present disclosure. Some aspects may comprise a computer program product for performing operations presented herein. For example, such a computer program product may comprise a computer-readable medium having stored thereon (and / or encoded thereon) instructions executable by one or more processors to perform operations described herein. In some aspects, the computer program product may include packaging materials.

[0116] Various components, modules, or units are described in this disclosure to highlight functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a hardware unit or provided by a collection of interoperable hardware units, including one or more processors as described above, along with appropriate software and / or firmware.

[0117] The terms "including," "comprising," and "having" and variations thereof mean "including but not limited to," unless otherwise specified.

[0118] Finally, the language used herein may have been selected primarily for ease of reading and instructional purposes, and not to delineate or limit the subject matter of the present invention. Accordingly, the scope of the technology is not intended to be limited by this detailed description, but rather by any claims that issue in an application based on this specification. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology described in the following claims.

Claims

1. A method (500) for data classification, comprising: Receiving (502) at least one first data set, the at least one first data set including at least one labeled data set and at least one unlabeled data set; processing (504) each of a plurality of first sub-datasets included in the at least one labeled dataset to generate at least one first meta-feature from the at least one labeled dataset, the at least one first meta-feature being at least one first cluster index calculated based on a plurality of sub-cluster indexes corresponding to each of the plurality of first sub-datasets; Associating (506) the at least one first meta-feature with a pre-constructed model (320) including a plurality of classification models, the pre-constructed model (320) further including at least one mapping function for mapping the at least one pre-computed meta-feature to a plurality of pre-computed classification performance scores corresponding to the plurality of classification models; estimating (508) a classification performance score for each of the plurality of classification models for the at least one labeled dataset based on associating the at least one first meta-feature with the pre-constructed model (320); and generating a list (510) including the plurality of classification models sorted in descending order of the estimated classification performance scores; selecting (512) a predetermined number of top classification models from the list to construct an ensemble classification model for classifying the at least one unlabeled dataset; A method comprising:

2. The method described in claim 1, wherein the at least one first meta-feature includes a plurality of first cluster indexes corresponding to each of a plurality of clustering models.

3. The method described in claim 2, wherein each of the plurality of first cluster indexes comprises an average value of the plurality of sub-cluster indexes of each of the plurality of first sub-datasets corresponding to each of the plurality of clustering models.

4. The method of claim 1, wherein the at least one first cluster index includes an external cluster index that requires a priori data to evaluate the results of the clustering model, and an internal cluster index that does not require a priori data to evaluate the results of the cluster model.

5. Classifying the at least one unlabeled dataset comprises: and processing the at least one unlabeled dataset using the ensemble classification model to predict a class label based on one of a majority vote, a weighted average, and model stacking.

6. Processing the at least one labeled dataset to generate at least one first meta-feature includes: processing the at least one labeled dataset to generate at least one cleaned dataset; processing the at least one cleaned data set using at least one clustering model to generate one or more clusters; generating a multi-dimensional vector by processing the one or more clusters, the multi-dimensional vector including the at least one first meta-feature; The method of claim 1 , comprising:

7. determining a classification complexity of the at least one first data set by comparing the estimated classification performance score to a pre-defined threshold; The method of claim 1 further comprising:

8. The pre-constructed model comprises: receiving at least one second data set; processing the at least one second data set to generate a plurality of training sub-data sets; processing the plurality of training sub-data sets using at least one clustering model to generate one or more clusters; generating a multi-dimensional vector by processing the one or more clusters, the multi-dimensional vector including at least one second meta-feature corresponding to the plurality of training sub-data sets, the at least one second meta-feature being at least one second cluster index calculated based on a plurality of training sub-cluster indexes corresponding to each of the plurality of training sub-data sets; generating a plurality of classification performance scores corresponding to the plurality of classification models by processing the plurality of training sub-data sets; generating the pre-constructed model by associating the generated at least one second meta-feature with the generated plurality of classification performance scores, the at least one second meta-feature corresponding to the at least one pre-computed meta-feature and the plurality of classification performance scores corresponding to the plurality of pre-computed classification performance scores; The method of claim 1 , wherein the compound is produced by 9. The method of claim 8, wherein the at least one second meta-feature includes a plurality of second cluster indexes corresponding to each of a plurality of clustering models.

10. The method of claim 9, wherein each of the plurality of second cluster indexes comprises an average value of the plurality of training sub-cluster indexes of each of the plurality of training sub-datasets corresponding to each of the plurality of clustering models.

11. The method of claim 8, wherein the at least one second cluster index includes an external cluster index that requires a priori data to evaluate the results of the clustering model, and an internal cluster index that does not require a priori data to evaluate the results of the cluster model.

12. 9. The method of claim 8, wherein generating a plurality of classification performance scores corresponding to the plurality of classification models comprises generating a best classification performance score for each of the plurality of classification models by tuning one or more hyperparameters corresponding to the plurality of classification models.

13. A system (120) for data classification, comprising: A memory (240); and at least one processor (230) communicatively coupled to the memory (240), the at least one processor (230) comprising: receiving at least one first data set, the at least one first data set including at least one labeled data set and at least one unlabeled data set; processing each of a plurality of first sub-datasets included in the at least one labeled dataset to generate at least one first meta-feature from the at least one labeled dataset, the at least one first meta-feature being at least a first cluster index calculated based on a plurality of sub-cluster indexes corresponding to each of the plurality of first sub-datasets; Associating the at least one first meta-feature with a pre-constructed model (320) including a plurality of classification models, the pre-constructed model (320) further including at least one mapping function for mapping the at least one pre-computed meta-feature to a plurality of pre-computed classification performance scores corresponding to the plurality of classification models; estimating a classification performance score of each of the plurality of classification models for the at least one labeled dataset based on associating the at least one first meta-feature with the pre-constructed model (320); generating a list including the plurality of classification models sorted in descending order of the estimated classification performance scores; selecting a predetermined number of top classification models from the list to construct an ensemble classification model for classifying the at least one unlabeled dataset; The system is configured as follows:

14. The at least one processor processes the at least one unlabeled dataset into: processing the at least one unlabeled dataset using the ensemble classification model to predict a class label based on one of a majority vote, a weighted average, and model stacking; The system of claim 13 configured for classification.

15. the at least one processor is configured to process the at least one labeled dataset to generate at least one first meta-feature; Generating the at least one first meta-feature comprises: processing the at least one labeled dataset to generate at least one cleaned dataset; processing the at least one cleaned data set using at least one clustering model to generate one or more clusters; generating a multi-dimensional vector by processing the one or more clusters, the multi-dimensional vector including the at least one first meta-feature; The system of claim 13 , further comprising:

16. The at least one processor 14. The system of claim 13, further configured to determine a classification complexity of the at least one first data set by comparing the estimated classification performance score to a preset threshold.

17. The at least one processor is further configured to generate the pre-constructed model; Generating the pre-constructed model includes: receiving at least one second data set; processing the at least one second data set to generate at least one training sub-data set; processing the at least one training sub-data set using at least one clustering model to generate one or more clusters; generating a multidimensional vector by processing the one or more clusters, the multidimensional vector including at least one second meta-feature corresponding to the at least one training sub-data set, the at least one second meta-feature being at least one second cluster index based on values ​​calculated for each of the at least one training sub-data set; generating a plurality of classification performance scores corresponding to the plurality of classification models by processing the at least one training sub-data set; generating the pre-constructed model by associating the generated at least one second meta-feature with the generated plurality of classification performance scores, the at least one second meta-feature corresponding to the at least one pre-computed meta-feature and the plurality of classification performance scores corresponding to the plurality of pre-computed classification performance scores; The system of claim 13 , wherein the pre-constructed model is generated by:

18. 20. The system of claim 17, wherein the at least one processor is configured to generate a plurality of classification performance scores corresponding to the plurality of classification models by tuning one or more hyperparameters corresponding to the plurality of classification models to generate a best classification performance score for each of the plurality of classification models.

19. 14. The system of claim 13, wherein the system is configured to provide a machine learning as a service (MLaaS) platform for data classification and classification model selection.

Images