METHOD AND SYSTEM FOR AUTOMATIC CONTROL OF MACHINE LEARNING MODELS

Abstract

The presented invention relates generally to the field of computer technology, and in particular to a method and system for the automatic control of machine learning models with the aim of simplifying the process of creating user-defined machine learning models, as well as increasing the speed of said process. The technical result, which the presented solution is aimed at achieving, is to provide the ability to automatically select a more accurate forecasting model, taking into account various sets of parameters required for forecasting by the models. The specified technical result is achieved by implementing a method for the automatic control of machine learning models, performed by at least one computing device, comprising the steps of: receiving from the user device a request for the provision of a forecasting model for performing a forecasting task;determine a list of models that can be used to perform a forecasting task, wherein the list of models contains at least: an identifier (id) of a first model and a first list of required parameters for forecasting; an id of a second model and a second list of required parameters for forecasting, wherein the second list of parameters contains at least one parameter different from the parameters from said first list of parameters; determine the presence of labeled data for the first list of parameters and the second list of parameters; determine a method for training or retraining models from the list of models; train or retrain the first model on the first list of parameters and the corresponding labeled data, and the second model on the second list of parameters and the corresponding labeled data; determine the quality indicators of the models from the list of models taking into account their corresponding lists of parameters required by the models for forecasting;determine the model with the highest quality score for the deployment of the trained model into production in the runtime environment.

Claims

1. A method for automatically controlling machine learning models, performed by at least one computing device, comprising the steps of: receive a request from the user's device to provide a forecasting model to perform the forecasting task; determining a list of models that can be used to perform the forecasting task, wherein the list of models contains at least: an identifier (id) of a first model and a first list of required parameters for forecasting; an id of a second model and a second list of required parameters for forecasting, wherein the second list of parameters contains at least one parameter different from the parameters from said first list of parameters; determine the presence of marked data for the first parameter list and the second parameter list; determine the methodology for training or retraining models from the list of models; carry out training or retraining of the first model on the first list of parameters and the corresponding labeled data, and the second model on the second list of parameters and the corresponding labeled data; determine the quality indicators of models from the list of models, taking into account the corresponding lists of parameters required by the models for forecasting; determine the model with the highest quality score for the release of the trained model into production operation in the runtime environment.

2. The method according to paragraph 1, characterized in that additional steps are performed in which it is determined that for at least one model from the list of models there is no labeled data, while training or further training of such a model is not carried out.

3. The method according to paragraph 1, characterized in that the following steps are additionally performed: analyze the quality indicators of models stored in the database; determine the presence of a third model, the quality indicators of which are higher than the indicators of the first and second models, for the operation of which a third list of parameters is required; determine the absence of at least one parameter from the third list of parameters in the available databases; request from the user's device at least one parameter that is not present in the available databases, and the model ID is included in the list of models only if information about the mentioned parameter that is not present in the available databases has been provided.

4. The method according to paragraph 1, characterized in that the determination of the quality indicators of the models is carried out taking into account the validation methodology determined for each model.

5. The method according to claim 1, characterized in that the list of models that can be used to perform the forecasting task is determined based on the task id.

6. The method according to paragraph 1, characterized in that the following steps are additionally performed: extract information about the required parameters for forecasting for each model; The presence of the required parameters in the available databases is checked, and the model ID is included in the list of models if all the required parameters are contained in the available databases.

7. The method according to paragraph 1, characterized in that the stage of determining the quality indicators of the models contains stages in which: feed each model from the list of models with the parameters required to predict these models in order to obtain the predicted results of the models; The predicted results obtained are compared with the actual results to determine the magnitude of the deviation of the models, with the highest quality score being assigned to the model having the lowest said deviation value.

8. The method according to paragraph 1, characterized in that the stage of determining the quality indicators of the models contains stages in which: for each model from the list of models, those parameters for which there are actual results are extracted from the set of parameters in relation to each object; feed the extracted parameters into the model input to obtain predicted results of the model's operation for each object; compare the predicted results with the actual results, and if the predicted result corresponds to the actual result, then assign a label to the predicted result of the model's operation indicating that the said predicted result corresponds to the actual result, and if the predicted result does not correspond to the actual result, then assign a label to the predicted result of the model's operation indicating that the said predicted result does not correspond to the actual result; determine the value characterizing the ratio of the marks indicating that the predicted result of the model's operation corresponds to the actual result to the marks indicating that the predicted result of the model's operation does not correspond to the actual result; The value obtained in the previous step is compared with the threshold value interval to determine the presence of a model risk; if the value obtained is outside the threshold value interval, a decision is made about the presence of a model risk and a minimum model quality indicator is assigned, and if the value does not go beyond the threshold value interval, a decision is made about the absence of a model risk and a maximum model quality indicator is assigned.

9. The method according to paragraph 1, characterized in that it additionally contains the steps of: determine that the quality indicators of the models are equal; The rating of the models is determined, and the model with the highest rating is selected for the release of the trained model into industrial operation in the runtime environment.

10. The method according to paragraph 1, characterized in that the stage of determining the quality indicators of the models contains stages in which: determine that the quality indicators of the models are equal; for each model, an analysis of the parameters from the list of parameters required for the model to operate is carried out to determine the number of data gaps; determines the ratio of the number of gaps to the total amount of data, which is assigned as an indicator of the quality of the parameters, and the model with the best quality indicators of the parameters is selected for the output of the trained model into industrial operation in the runtime environment.

11. The method according to paragraph 1, characterized in that the stage of determining the quality indicators of the models contains stages in which: determine that the quality indicators of the models are equal; The quality indicator of the parameters is determined for each model based on the data format, and the model with the best quality indicators of the parameters is selected for the release of the trained model into industrial operation in the runtime environment.

12. The method according to paragraph 1, characterized in that the stage of determining the quality indicators of the models contains stages in which: determine that the quality indicators of the models are equal; determine for each model the amount of data containing the parameters required for the model to operate; Based on the amount of data, a rating is assigned to the model, with the highest rating being assigned to the model that requires the least amount of data, and the model with the highest rating is deployed into production in the runtime environment.

13. A system for automatic control of machine learning models, comprising at least one computing device and at least one memory device containing machine-readable instructions that, when executed by at least one computing device, perform the method according to any one of paragraphs 1-12.

Images