Method and system for automatically creating a machine learning model

Abstract

The claimed technical solution relates generally to the field of machine learning, and more particularly to a method and system for automatically creating a machine learning model for solving classification problems oriented toward real-world problems. The technical result of the claimed method is the automatic creation of a machine learning model that has low resource requirements but a high generalization ability and, consequently, the ability to make high-quality predictions of target events. This technical result is achieved by means of the present method for automatically creating a machine learning model for solving a classification problem, which is implemented using at least one computing device and contains the steps of: setting parameter values; creating scripts; producing a training dataset; determining the architecture of a model and a list of features of the model; generating a machine learning model; calculating the classification metrics of the machine learning model.

Description

METHOD AND SYSTEM FOR AUTOMATIC CREATION OF A MACHINE LEARNING MODEL AREA OF TECHNOLOGY

[0001] The claimed technical solution generally relates to the field of machine learning, and in particular to a method and system for automatically creating a machine learning model for solving classification problems aimed at real-world problems. LEVEL OF TECHNOLOGY

[0002] Currently, the creation and implementation of machine learning models in various fields of life has become an integral part of these industries. Machine learning algorithms are used to recognize voices, images, process natural language texts, and much more. The implementation of machine learning models has made a real revolution in many fields of human activity.

[0003] There are many ways to create and train machine learning models to perform assigned tasks, such as predicting the occurrence of a target event, etc. The general principle of the model training process is to pre-process the data to prepare for the training process, determine the characteristic features of the generated training data set, construct new features from the specified set, optimize the hyperparameters of the machine learning model and the training process itself of the generated model with optimal parameters.

[0004] However, this principle is highly susceptible to the influence of the human factor, since the specified stages of the training process are performed by a specialist in this field of technology. In addition, manual training of models is a lengthy process and, in certain situations, even ineffective.

[0005] The prior art also includes approaches based on the principle of automatic machine learning (AutoML) – the process of automating the end-to-end process of applying machine learning to real-world problems.

[0006] Thus, the prior art discloses a solution disclosed in application No. US 2020159690 Al (SAP SE [DE]), published on 21.05.2020. The said solution, in particular, discloses a system for assessing indicators using a classification approach with automatic machine learning, designed with the ability to receive and pre-process a data set, generate a configuration file for creating a machine learning model, select the required model from template machine learning models by enumerating the specified templates for the data type and calculating their accuracy.

[0007] The disadvantage of this solution is the low speed of generating a machine learning model due to the lengthy procedure of sorting through template models for the classification problem being solved. In addition, this process of defining a model and selecting a suitable model from the presented templates also increases the resource intensity of the machine learning model, since the best of the presented models is selected, which, obviously, can have an extremely high resource intensity (number of parameters), which, as a consequence, leads to insufficient quality (accuracy) for obtaining correct results.

[0008] Common drawbacks of existing solutions are the lack of an effective way to automatically create a machine learning model to solve real-world problems, ensuring low resource intensity and high speed of model generation while maintaining high quality (accuracy) to obtain correct results. DISCLOSURE OF INVENTION

[0009] The declared technical solution proposes a new approach to the automatic creation of a machine learning model for solving the classification problem. This solution uses automatic generation of a machine learning model based on the automatic selection of the optimal architecture and feature space used to train the model.

[0010] Thus, the technical problem of reducing resource intensity and increasing the speed of model generation while maintaining high model accuracy is solved. [UN] The main technical result that emerges from solving the above problem is an increase in the accuracy of forecasting the target event by increasing the accuracy of the generated machine learning model.

[0012] An additional technical result that appears when solving the above problem is a reduction in resource intensity and an increase in the speed of model generation.

[0013] The specified technical results are achieved by implementing a method for automatically creating a machine learning model for solving a classification problem, performed by at least one computing device, comprising the following stages: a) obtaining a first data set for training in accordance with the classification problem being solved, wherein said data set contains at least: a data collection period and a set of feature space data including categorical and numerical features for the classification problem being solved; b) obtaining a second data set for training, containing data on the occurrence of a target event of the classification problem being solved; c) forming a representative sample for training the model based on the first and second data sets; d) determining the architecture of the machine learning model, during which: i. setting the quality parameter of the model; ii. obtaining the initial number of trees of the model; iii.training the model on the data obtained in steps a) and b) with a given initial number of trees and comparing the quality parameter of the model with the given quality parameter of the model; iv. repeating step iii, recursively reducing the number of trees, until the stopping criterion; v. generating a file with the model architecture based on the number of trees obtained in step iv; e) determining the number of essential features for the machine learning model, wherein during the determination of said features: i. obtaining the architecture of the machine learning model determined in step d) and the quality parameter of the model; ii. obtaining a data set of the feature space; iii. excluding at least one feature from the set and training the model; iv. determining the quality of the model and comparing it with a given model quality parameter; v. repeating steps ii-iii until the stopping criterion is reached; vi. forming a set of essential features based on the data obtained in step v; f) generating a machine learning model based on the data obtained in steps d) and e) and training the said model using a representative sample obtained in step c); g) calibrating the model obtained in step f); h) processing the data using the machine learning model obtained in step f), during which the probability of occurrence of the target event for the classification problem being solved is obtained; i) performing an action corresponding to the results obtained in step h).

[0014] In another particular embodiment of the method, the first data set is a representative data set.

[0015] In another particular embodiment of the method, the categorical features represent at least the following features: gender, marital status, city of residence.

[0016] In another particular embodiment of the method, the numerical features represent at least the following features: age, number of previous requests.

[0017] In another particular embodiment of the method, the stopping criterion for determining the architecture of the machine learning model is at least a parameter characterizing the minimum number of trees of the machine learning model, while maintaining the quality parameter of the machine learning model above the threshold quality parameter.

[0018] In another particular embodiment of the method, the stopping criterion for determining essential features is at least a threshold parameter as a model.

[0019] In another particular embodiment of the method, the classification problem being solved represents at least the probability of a client calling the call center.

[0020] In another particular embodiment of the method, the action corresponding to the classification problem being solved is the distribution of the load between the equipment servicing the call processing center.

[0021] In addition, the stated technical results are achieved through a system for automatically creating a machine learning model for solving a classification problem, containing: • at least one processor; • at least one memory connected to the processor, which contains machine-readable instructions that, when executed by at least one processor, ensure the execution of a method for automatically creating a machine learning model for solving a classification problem. BRIEF DESCRIPTION OF DRAWINGS

[0022] The features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings.

[0023] Fig. 1 illustrates a block diagram of the implementation of the claimed method.

[0024] Fig. 2 illustrates an example of the general appearance of a computing system that ensures the implementation of the claimed solution. IMPLEMENTATION OF THE INVENTION

[0025] Below we will describe the concepts and terms necessary for understanding this technical solution.

[0026] A model in machine learning (ML) is a set of artificial intelligence methods, the characteristic feature of which is not a direct solution to a problem, but learning in the process of applying solutions to many similar problems.

[0027] The classification problem in machine learning is the problem of assigning an object to one of the predetermined classes based on its formalized features.

[0028] Gradient boosting is a machine learning technique used in regression and classification problems, among others. It produces a predictive model as a collection of weak predictive models, which are typically decision trees.

[0029] The Gini coefficient is a quality metric that is often used to evaluate predictive models in binary classification problems.

[0030] The claimed technical solution offers a new approach that ensures a decrease in resource intensity and an increase in the speed of generating machine learning models for solving classification problems without a significant decrease in the generalizing ability aimed at real-world problems, for example, the tasks of predicting the occurrence of a target event. In addition, the claimed solution offers an approach that ensures the ability to automatically generate machine learning models, which, accordingly, eliminates the influence of the human factor on the learning process. Another advantage that appears when using the claimed invention is a decrease in the computing power required to create and train models due to the effective process of setting up the architecture and features used in the models that solve the tasks.

[0031] This technical solution can be implemented on a computer, in the form of an automatic system containing input / output interfaces, or a machine-readable medium containing instructions for performing the above-mentioned method.

[0032] The technical solution can be implemented as a distributed computer system.

[0033] In this solution, the term “system” refers to a computer system, a computer (electronic computer), a numerical control (CNC), a PLC (programmable logic controller), computerized control systems, and any other devices capable of performing a given, clearly defined sequence of computing operations (actions, instructions).

[0034] A command processing unit is an electronic unit or integrated circuit (microprocessor) that executes machine instructions (programs) /

[0035] The instruction processing unit reads and executes machine instructions (programs) from one or more storage devices, such as random access memory (RAM) and / or read-only memory (ROM). ROM may be, but is not limited to, limited to hard disk drives (HDD), flash memory, solid state drives (SSD), optical storage media (CD, DVD, BD, MD, etc.), etc.

[0036] A program is a sequence of instructions intended for execution by a computer control device or command processing device.

[0037] The term "instructions" as used in this application may refer generally to software instructions or software commands that are written in a given programming language to perform a specific function, such as, for example, obtaining hardware and software solution artifacts, generating a digital standard for a hardware and software solution, generating software and hardware solution verification results, analyzing data, etc. The instructions may be implemented in a variety of ways, including, for example, object-oriented methods. For example, the instructions may be implemented using the C++ programming language, Java, Python, various libraries (e.g., MFC; Microsoft Foundation Classes), etc. The instructions that perform the processes described in this solution may be transmitted via both wired and wireless data transmission channels, such as Wi-Fi, Bluetooth, USB, WLAN, LAN, etc.

[0038] Fig. 1 shows a block diagram of a method 100 for automatically creating a machine learning model for solving a classification problem, which is disclosed step by step in more detail below. Said method 100 consists of performing steps aimed at processing various digital data. The processing is typically performed using a system that can represent, for example, a server, a computer, a mobile device, a computing device, etc. The elements of the system are disclosed in more detail in Fig. 2.

[0039] At step 110, a first set of training data is obtained in accordance with the classification problem to be solved, wherein said set of data contains at least: a data collection period and a set of feature space data including categorical and numerical features for the classification problem to be solved.

[0040] At the specified step 110, the system 200 receives input data for solving the classification problem. Thus, the process of obtaining data can be implemented by means of the system 200 accessing data repositories, for example, internal databases that contain data on the target event. As The data storages can be either specific clients (all data is stored in CSV files), or SQL servers, etc. The access to said storages can be carried out via a data transmission network, for example, the Internet, LAN, etc. In one particular embodiment, the data can be loaded into the system using an input / output interface, for example, by a user of the system 200.

[0041] The input data set is a set of historical data on the target event corresponding to the classification task being solved, sufficient for creating and training the machine learning model. Thus, in one particular embodiment, the user of the system 200 selects the type of task that needs to be solved using the claimed method. In accordance with the selected type of task, the system 200 receives a corresponding set of historical data associated with this task.

[0042] For example, classification tasks related to real-world problems may include such tasks as determining the probability of a customer calling a call center in the next month, the probability of a customer visiting a bank branch in a certain period, the probability of a person seeking medical care, forecasting the load on a city's transport system, etc. Each of these tasks affects the infrastructure associated with the organization servicing the presented area of ​​life. Thus, if we are talking about the probability of a customer calling a call center, then the problem in this area will be the load on the equipment, i.e. the fault tolerance of the infrastructure, the failure or malfunction of which, as a result, can lead to a decrease in security (the inability to prevent threats).A similar situation may arise in medical institutions, where forecasting the flow of people directly affects the speed of medical care and the number of medical personnel required to provide such care in a timely manner. As will be obvious to a specialist in this field of technology, the classification task in this solution is a general term reflecting the process of determining the probability of the occurrence of a target event, on the basis of which a target action is performed, respectively, the classification task in this solution is understood as the task of determining the occurrence of a target event based on its historical data.

[0043] It is worth noting that in particular cases of the implementation of the claimed invention, the classification problem is reduced to a binary classification problem, i.e. the event either occurred or did not. In addition, an important criterion for solving the problem is the availability of historical data. For example, it is impossible to solve the problem of predicting the probability of a customer calling the contact center next month without having historical data on his calls.

[0044] Thus, for example, at step 110 the input data set can be obtained by connecting to a database, for example, a SQL server. Thus, to implement this process, the user, in the GUI of the system, for example, system 200, can set the following set of parameters: the SQL server connection configuration parameter; the unique model identifier parameter; the model training date parameter; the model validation date parameter; the parameter of the list of features to exclude from model training. The specified parameters, in essence, provide the ability to create and allocate memory and power by the computing device for implementing further stages of method 200.

[0045] Thus, at step 110, the system 200 receives a first data set containing a data collection period for the classification problem being solved and a feature space data set including categorical and numerical features for the classification problem being solved. The data collection period may depend both directly on the required horizon for determining the target event and on the amount of historical data stored in the data warehouse. Thus, the data collection period may be 180 days, 365 days, etc. In addition, in another particular embodiment, the data collection period may depend on the amount of historical data, for example, a minimally sufficient set of historical data (a representative set of data).Thus, a representative data set is a sample in which all the main features of the general population from which the given sample is extracted are presented approximately in the same proportion or with the same frequency with which the given feature appears in this general population. The specified criterion can be verified by the system 200, for example, at the stage of selecting a data warehouse, for example, by the number of historical data stored (10,000 records, etc.). Data on the number of records of each section of the database can be stored and determined from the data table file, for example, from metadata.

[0046] In this solution, the attribute space is understood as a set of attributes that reflect the properties of an object with sufficient completeness. Thus, the attribute space in one particular embodiment includes sets of categorical features of the object and numerical features of the object. Categorical features are at least gender, city of residence, city of birth, previous illnesses, type of vehicle, etc. Numerical features are at least age, number of calls to the organization for the selected period, number of vehicles, etc. That is, the feature space characterizes the most complete and objective information about the properties of the object under study. Obviously, depending on the classification problem being solved, the feature space of the object will differ. For example, to predict the probability of a call to the contact center, the first set of training data will contain the following categorical and numerical features: number of calls in the last three months, gender, age, place of birth, place of residence.

[0047] Thus, at step 110, a first set of training data is obtained in accordance with the classification problem being solved, wherein said set of data contains at least: a data collection period and a set of feature space data including categorical and numerical features for the classification problem being solved.

[0048] Next, at step 120, a second set of training data is obtained, containing data on the occurrence of the target event of the classification problem being solved.

[0049] At the specified step 120, the system 200 receives data on the occurrence or non-occurrence of the target event. Thus, the second set of data for training is also formed on the basis of historical data and contains data on whether the event occurred. The specified set of data can be obtained, as well as the first set of data, via a data transmission network, for example, via the Internet. Data on the occurrence of the target event can be stored both in a separate database and in a database column containing data for the first set of training data formed at step 110. Thus, to predict the probability of a call to the contact center, data on the fact of the occurrence or non-occurrence of the target event are data on a call made or not made by a certain client and / or patient.

[0050] Thus, in one particular embodiment, at step 120, for generating the second set of data, for example, a unique model identifier; unique identifier of the target event; data collection date; fact of occurrence of the target event;

[0051] In addition, in another particular embodiment, a third data set may additionally be created, containing data on the feature space of the target event of the classification problem being solved. Said set is intended to accelerate the process of training the machine learning model. Thus, said set may be formed using the representativeness filter parameter, i.e. a filter reflecting relevant features for training the model. Thus, in another particular embodiment, the third data set may contain a data set of the feature space of the target event, including categorical and numerical features for the classification problem being solved.

[0052] In another particular embodiment, the data on the occurrence of the target event may be data on equipment failure with a different number of calls. The authors of the present invention note that, although the problem of contact center fault tolerance is given as an example, the presented invention can also be applied in other areas of life, for example, in a room climate control system. Thus, the data on the occurrence of the target event in this case will be data on whether or not the heating system is turned on when a certain temperature is reached.

[0053] At step 130, a representative sample is formed for training the model based on the first and second sets of data.

[0054] At step 130, the first and second data sets are processed to obtain a representative sample for training the machine learning model. A representative sample is one whose Target Rate matches the Target Rate of the General Population.

[0055] Thus, data on the fact of the target event occurrence are taken from the second data set. The specified data are compared with the data from the first set. The specified process can be implemented, for example, by means of a procedure for comparing and searching through data tables, for example, a search tool in text files. After finding relevant (representative values), the specified values ​​are compared with each other and saved in a new data file.

[0056] In another particular embodiment, a representative sample for training the model may be pre-formed and sent to the system 200, for example, by a user of the system 200.

[0057] Thus, at the specified stage 130, a representative sample is formed for training the model based on the first and second sets of data.

[0058] Next, method 100 proceeds to step 140.

[0059] At step 140, the architecture of the machine learning model is determined. The architecture of the model is determined in several sub-steps, during which: the quality parameter of the model is set; the initial number of trees of the model is obtained; the model is trained on the data obtained at steps 110 and 120, with a given initial number of trees, and the quality parameter of the model is compared with the given quality parameter of the model; the stage of training the model is repeated, recursively reducing the number of trees, up to the stopping criterion; a file with the architecture of the model is formed based on the number of trees obtained after stopping the algorithm.

[0060] Let's take a closer look at this process of defining the architecture of a machine learning model.

[0061] Thus, the definition of the architecture of the machine learning model is understood as the automatic adjustment and selection of the number of decision trees of the machine learning model, which provide the maximum generalizing ability to predict the target event. For this, at the specified stage 140, the quality of the model is set. The quality of the model, in one particular embodiment, is determined by the parameter - the Gini coefficient. The specified coefficient reflects the share of the decrease in the quality of the final model compared to the quality of the best model, i.e. the maximum possible value of the quality of the model. The specified parameter, in one particular embodiment, can be set manually by the user of the system. In another particular embodiment, the Gini coefficient can be pre-set in the system 200, and can be equal to, for example, 99.5%. The quality of the model in this solution should be understood as the ability of the model to correctly predict a given class in solving the classification problem.

[0062] The determination of the architecture of the machine learning model is performed automatically by the system 200, i.e. the user of the system is not involved in the process. The machine learning model, in one particular embodiment, is a gradient boosting model, such as the LightGBM model, etc. In more detail, defining the architecture of a machine learning model consists of tuning its parameters. Thus, most machine learning algorithms include a large number of parameters, including regularization parameters, stochastic gradient descent parameters, and other parameters such as the maximum number of trees. The model parameters have a significant impact on the prediction accuracy of the resulting models without clear values ​​applicable to different data sets. The traditional way to find the parameters is manual tuning. The accuracy of the generated machine learning model depends on the minimum prediction error. Thus, in one particular embodiment, the parameter of the machine learning model is the number of trees parameter.

[0063] To determine the said parameter, and therefore to determine the architecture of the model, the system 200 in the next step sets the initial number of trees, for example, equal to 5000 trees. It will be obvious to a person skilled in the art that the initial parameter of the number of trees can be set to any integer, positive value.

[0064] After that, the training of the presented model configuration is performed, during which the data obtained in step 110 and the data obtained in step 120 are processed by the model to obtain the model forecasting results. Thus, the LIGhtGBm model can be used as a model (the model is described in more detail at the link, for example: https: / / lightgbm.readthedocs.io / en / latest / ).

[0065] After training a given model configuration containing a given number of trees, the quality of said model is compared with a threshold quality parameter. As indicated above, the quality of the model can be compared using the Gini coefficient. If the quality parameter satisfies the given one, then the algorithm terminates. If the quality parameter is lower than the given one, then the system 200 recursively reduces the initial number of trees, and repeats the training sub-stage with the new number of trees. This process continues until a stopping criterion is reached. In one particular embodiment, the stopping criterion is at least a parameter characterizing the minimum number of trees of the machine learning model, while maintaining the quality parameter of the machine learning model above the threshold quality parameter. It is worth It should be noted that the stopping criterion at said step 140 is referred to in the context of the stopping criterion for determining the architecture of the machine learning model and can also be called the first stopping criterion.

[0066] After reaching the stopping criterion (the first stopping criterion), the machine learning model configuration file is saved in the system memory 200. The final model configuration file contains the configured machine learning model architecture for the classification problem being solved, for example, for predicting the probability of a call to a contact center. In this case, in accordance with the stopping criterion, the algorithm selects the most optimal number of trees while maintaining the quality of the model. As noted above, the number of trees also affects the speed of model training.

[0067] Thus, at the specified stage 140, the architecture of the machine learning model is determined, during which a configuration file is formed containing the number of decision trees of the model.

[0068] Next, at step 150, the number of significant features for the machine learning model is determined.

[0069] Thus, training data sets contain many features, however, not all features affect the accuracy of the machine learning model forecasting, while such features can be quite voluminous, and, as a result, lead to a decrease in the speed of model training and an increase in the computing resources required for training. This feature of the claimed solution allows increasing the speed of training by automatically determining essential features from the general set of feature space.For this purpose, at the specified step 150, the following steps of the algorithm are performed: the architecture of the machine learning model determined at step 140 and the model quality parameter are obtained; a set of feature space data are obtained; at least one feature is excluded from the set and the model is trained; the quality of the model is determined and compared with the threshold model quality parameter; the process is repeated until the stopping criterion is reached; a set of essential features is formed based on reaching the stopping criterion.

[0070] Let's take a closer look at these steps.

[0071] In the first step of said stage 150, the model architecture from the configuration file from stage 140 and the quality threshold criterion (Gini coefficient) are extracted by the system 200. For example, the configuration file can be extracted from the internal memory of the system 200. Accordingly, the Gini coefficient can also be obtained from the previous step 140.

[0072] The next step is to obtain a set of feature space data. The set, as indicated above, was obtained at step 110 and contains all categorical and numerical features characterizing the classification problem being solved. For example, categorical features may be gender, marital status, type of employment, dual citizenship, etc. That is, the number of features may be quite large, while it is obvious that most of these features will not relate to a specific classification problem. Accordingly, numerical features may also contain a large amount of irrelevant data. The following actions are performed to determine the essential features.

[0073] After receiving a set of feature space data, at least one feature is excluded from the set and the model is trained.

[0074] Next, the quality of the model is determined and compared with the threshold parameter of the model quality. If the quality satisfies the specified parameter, then the previous step is repeated and one more feature is excluded from the feature space. Thus, when some features are excluded, the quality of the model may even increase, in connection with which it is necessary to achieve the minimum possible set of features sufficient for training a machine learning model with high generalizing ability. Accordingly, the specified procedure for excluding features and determining the current quality of the model is repeated until the stopping criterion is reached. The stopping criterion is at least the threshold parameter of the model quality and is the stopping criterion for determining the essential features, which can also be referred to in this technical solution as the second stopping criterion.

[0075] Thus, in one particular embodiment, the system 200 performs a sequential exclusion of features from the model, during which: each feature from the representative sample is recursively excluded; a model with a given value of the first parameter of the finite number of trees is created; the model is trained; the quality of the model is determined. These steps are repeated for each feature. After that, the excluded features are sorted in descending order of model quality. Next, the system 200 receives the very first feature from the sorted list and compares the quality of the model without this feature with threshold parameter of the model quality. If the stopping criterion is not met, the given feature is discarded. After that, the process is repeated.

[0076] Thus, at the specified step 150, a set of essential features for training a machine learning model is determined from a set of feature space data.

[0077] Next, at step 160, a machine learning model is generated based on the data obtained at steps 140 and 150, and said model is trained using the representative sample obtained at step 130.

[0078] The final machine learning model consists of the architecture obtained in step 140 and the salient features obtained in step 150.

[0079] As stated above, the model is an LGBM model. After the final model is formed, the training of the specified model is performed. The training of the model is performed on the data set obtained in step 130.

[0080] Thus, in one particular implementation, the model was trained to determine the probability of a customer calling the Bank in the following month after scoring to provide the necessary computing resources to ensure the smooth operation of the customer call center.

[0081] Thus, active clients (users) were used to train the model. Active clients can be obtained from the call database stored in the organization. An active client is one who has made a call to the center during a predetermined period, for example, a month, a week, etc. Then, from the specified clients, clients (users) were selected who, in the next period, after receiving the data set with active clients, for example, in the next month, made a call to the Bank. All other clients were marked as inactive and did not belong to the target. It is worth noting that the formation of the training sample was performed automatically, i.e. after the system received the input data set, the system, using, for example, algorithms for searching and comparing data in databases, compared active clients and clients who made a call in the next period. The feature space for training consisted of 500 features.

[0082] At the first stage, the LGBMClassifier() model was trained with the number of trees equal to 105. 98 features were above the information content threshold.

[0083] At the second stage, the LGBMClassifier() model was trained with the number of trees equal to 1. After an iterative increase in the number of trees, the stopping criterion was met on 51 trees, while the GINI metric became equal to 60.3%. 18 features were above the information content threshold.

[0084] At the third stage, the LGBMClassifier() model was trained with the number of trees equal to 51. After iteratively discarding the feature from the model, the GINI metric became equal to 59.12%, while 6 features remained in the model.

[0085] At the fourth stage, the LGBMClassifier() model was trained with the number of trees equal to 1. After an iterative increase in the number of trees, the stopping criterion was met on 31 trees, and the GINI metric became equal to 59.04%. As a result, the final model included the following features: 1. Number of incoming calls to the Bank over the last month 2. Number of outgoing contacts with the client over the last month 3. Number of logins to the mobile application over the last month 4. The amount of receipts to the client's accounts for the last month 5. Client income 6. Total balance on the client's accounts Thus, the model uses both information on the client’s communication with the Bank and his funds in the Bank’s accounts.

[0086] It is worth noting that the final feature space, depending on the classification task being solved, will contain its own unique set of features. Accordingly, the specified features are provided solely for understanding the principle of implementing the declared technical solution and are not intended to limit the implementation options.

[0087] At step 170, after training the model, calibration of the model obtained at step 160 is also performed. For example, calibration of the final model can be performed using the Isotonic Regression model.

[0088] Let us consider this process in more detail.

[0089] In the stated solution, the quality of the model was measured using the Gini metric obtained on the validation dataset. The dataset for validating the final model was automatically configured using the following algorithm: data from the list of the feature space of the final model from the next month are taken and scaled with using the SQL script of the final model. The resulting client score is joined with data on the occurrence of the target event in the month following the scoring. The classification metrics obtained on the data from the month of training and the data obtained from the month following the training are given in Table 1. Table 1. Distribution of clients in the Train and OOT samples. Data for Training Data for Validation Month 2022-11-30 2022-12-31 Total clients 9118 9215 Target clients 659,594 Netarget clients 8459 8620 Target Rate, share 7.2% 6.5% GINI 59.04% 58.76% The results of the Gini metric comparison show that the claimed method does not significantly reduce the metric over time.

[0090] At step 180, the data is processed using the machine learning model obtained at step 160, during which the probability of the target event for the classification problem being solved is obtained. Thus, in the claimed solution, in one particular embodiment, a set of data to be processed is sent to the trained machine learning model. For example, the data processed using the said model, in one particular embodiment, is, for example, data on 50,000 active clients. As mentioned above, active clients are users who have made a specified number of calls to the contact center, for example, at least 1 call per month. It is worth noting that the type of data processed by the generated model always matches the training data. That is, if we are talking about predicting the load on the contact center, then it is necessary to determine the number of users.Similar requirements are met for other tasks where classification is required (e.g., reception department, etc.). Next, for the received data, the model produces a call probability parameter. Based on this parameter, actions related to the specified data type are performed. In addition, the value of customer call probabilities can also be used when forming the staff of employees servicing the contact center.

[0091] Thus, at step 180, data processing is carried out, during which the probability of the occurrence of the target event for the classification problem being solved is obtained.

[0092] At step 190, an action is performed corresponding to the results obtained at step 180.

[0093] At the specified step 190, the result of determining the probability of occurrence of the target event of the classification problem being solved is sent to the decision-making system. The decision-making system may be, for example, a server, a computer, a tablet, etc., and is designed with the ability to perform actions based on the prediction received. For example, such actions may be proactive actions related to preventing the overload of the organization's infrastructure. Thus, continuing the example of the probability of a call to the contact center, the decision-making system is designed with the ability to calculate the density of the call flow, based on the data received at step 180, and compare the density indicators with the indicators of the equipment servicing the said calls. If the density indicator exceeds the permissible value, the load on the contact center can be distributed between other centers.

[0094] Thus, the specified application materials disclose a method for automatically creating a machine learning model for solving a classification problem.

[0095] Fig. 2 shows an example of a general view of a computing system 200 that ensures the implementation of the claimed method or is part of a computer system, for example, a server, a personal computer, part of a computing cluster, processing the necessary data for implementing the claimed technical solution.

[0096] In general, the system 200 comprises components such as: one or more processors 201, at least one memory 202, data storage means 203, input / output interfaces 204, I / O means 205, network interaction means 206, which are connected via a universal bus.

[0097] The processor 201 performs the basic computing operations necessary for processing the data when performing the method 100. The processor 201 executes the necessary machine-readable instructions contained in the RAM 202.

[0098] Memory 202 is usually implemented as RAM and contains the necessary software logic to provide the required functionality.

[0099] Data storage means 203 can be implemented in the form of HDD, SSD disks, RAID array, flash memory, optical storage devices (CD, DVD, MD, Blue-Ray disks), etc. Means 203 allow long-term storage of various types of information, such as transaction request processing history (logs), user identifiers, etc.

[0100] To organize the operation of the components of the 200 system and to organize the operation of external connected devices, various types of I / O interfaces 204 are used. The choice of the corresponding interfaces depends on the specific design of the computing device, which may be, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0101] The choice of interfaces 204 depends on the specific implementation of system 200, which can be implemented on the basis of a wide range of devices, for example, a personal computer, mainframe, laptop, server cluster, thin client, smartphone, server, etc.

[0102] The following may be used as I / O data means 205: keyboard, joystick, display (touch display), monitor, touch display, touchpad, mouse, light pen, stylus, touch panel, trackball, speakers, microphone, augmented reality means, optical sensors, tablet, light indicators, projector, camera, biometric identification means (retina scanner, fingerprint scanner, voice recognition module), etc.

[0103] Network interaction means 206 are selected from devices that provide network reception and transmission of data, for example, an Ethernet card, a WLAN / Wi-Fi module, a Bluetooth module, a BLE module, an NFC module, IrDa, an RFID module, a GSM modem, etc. With the help of means 205, data exchange is organized between, for example, a system 200, represented in the form of a server and a data storage containing the first and second sets of data for training, via a wired or wireless data transmission channel, for example, a WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM.

[0104] The specific selection of elements of system 200 for the implementation of various software and hardware architectural solutions may vary while maintaining the required functionality provided.

[0105] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology. Thus, the scope of the present technical solution is limited only by the scope of the attached formula.

Claims

FORMULA 1. A method for automatically creating a machine learning model for solving a classification problem, which is performed by at least one computing device and comprises the steps of: a) obtaining a first data set for training in accordance with the classification problem being solved, wherein said data set contains at least: a data collection period and a feature space data set including categorical and numerical features for the classification problem being solved; b) obtaining a second data set for training, which contains data on the occurrence of a target event of the classification problem being solved; c) forming a representative sample for training the model based on the first and second data sets; d) determining the architecture of the machine learning model, during which: i. setting the quality parameter of the model; ii. obtaining the initial number of trees of the model; iii.: training the model on the data obtained in steps a) and b) with a given initial number of trees, and comparing the model quality parameter with a threshold model quality parameter; iv. repeating step iii, recursively reducing the number of trees until a stopping criterion; v. generating a file with the model architecture based on the number of trees obtained in step iv; e) determining the number of essential features for the machine learning model, wherein the determination of said features comprises: i. obtaining the machine learning model architecture determined in step d) and the model quality parameter; ii. obtaining a data set of feature space; iii. excluding at least one feature from the set and training the model; iv. determining the quality of the model and comparing it with the threshold model quality parameter; 22 v. repeating steps ii-iii until the stopping criterion is reached; vi. forming a set of essential features based on the data obtained in step v; f) generating a machine learning model based on the data obtained in steps d) and e) and training the said model using the representative sample obtained in step c); g) calibrating the model obtained in step f); h) processing the data using the machine learning model obtained in step f), which results in obtaining the probability of occurrence of the target event for the classification problem being solved; i) performing an action corresponding to the results obtained in step h).

2. The method according to claim 1, characterized in that the first data set is a representative data set.

3. The method according to paragraph 1, characterized in that the categorical features represent at least the following features: gender, marital status, city of residence.

4. The method according to paragraph 1, characterized in that the numerical features represent at least the following features: age, number of user requests.

5. The method according to claim 1, characterized in that the stopping criterion for determining the architecture of the machine learning model is at least a parameter characterizing the minimum number of trees of the machine learning model, while maintaining the quality parameter of the machine learning model above the threshold quality parameter.

6. The method according to paragraph 1, characterized in that the stopping criterion for determining essential features is at least a threshold parameter as a model.

7. The method according to paragraph 1, characterized in that the classification problem being solved represents at least the probability of a client calling a call center.

8. The method according to paragraph 1, characterized in that the action corresponding to the classification problem being solved is the distribution of the load between the equipment servicing the call processing center. A system for automatically creating a machine learning model for solving a classification problem, containing: • at least one processor; • at least one memory connected to the processor, which contains machine-readable instructions that, when executed by at least one processor, ensure the execution of the method according to any one of claims 1-8.

Images