Machine learning platform-based model detection method, system and equipment

Abstract

The invention provides a model detection method, system and device based on a machine learning platform, and the method comprises the steps: obtaining model detection data which comprises prediction data, a prediction result obtained after the prediction data are inputted into a to-be-detected model, and evaluation data corresponding to the prediction data; under the condition that the prediction result and / or the evaluation data meet a preset model detection condition, performing feature analysis based on the prediction data, and determining a feature analysis result; determining a model index evaluation result of the to-be-tested model based on the evaluation data; and determining a detection result based on the to-be-detected model deployed in the machine learning platform according to the feature analysis result and the model index evaluation result. By introducing the evaluation data, the actual data support is increased, the detection effect is enhanced, the detection result is finally determined according to the feature analysis result and the model index evaluation result, and compared with manual monitoring, the accuracy is higher, and the detection result is more persuasive.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of model monitoring, and in particular, to a method, system, and device for model detection based on a machine learning platform. Background Art

[0002] After the trained model is launched into the production environment, over time, due to the change in the distribution of prediction data, model drift occurs, and the performance of the model will decline. Therefore, it is still necessary to monitor the performance of the launched model, so as to retrain the model or select a model with better performance from the candidate models for launch.

[0003] However, at present, most of the judgments on whether the model drifts are made manually based on experience, and it is impossible to make real-time model drift judgments based on actual data and model performance indicators. Moreover, in actual business, abnormal prediction data, or changes in the distributions of feature variables and target variables may all lead to model drift. When model drift occurs, manual judgment cannot quantify the degree of drift and explain the reasons for drift, and cannot provide available bases and effective suggestions for retraining the model. Further, the current monitoring only monitors the performance of the online model and cannot simultaneously monitor multiple models such as the online model and other candidate models. When the online model drifts, the corresponding processing mechanism cannot be triggered, and the online model cannot be replaced. Summary of the Invention

[0004] The embodiments of the present invention provide a method, system, and device for model detection based on a machine learning platform to solve the problem of inaccurate judgment caused by the existing manual monitoring of whether the model drifts.

[0005] To solve the above technical problems, the present invention is implemented as follows:

[0006] In a first aspect, the embodiments of the present invention provide a method for model detection based on a machine learning platform, including:

[0007] Obtain model detection data, where the model detection data includes prediction data, a prediction result obtained after the prediction data is input into the model to be tested, and evaluation data corresponding to the prediction data, where the evaluation data is the result obtained by applying the prediction data to actual business;

[0008] When the prediction result and / or the evaluation data meet the preset model detection conditions, perform feature analysis based on the prediction data to determine the feature analysis result;

[0009] Based on the evaluation data, determine the model index evaluation result of the model to be tested;

[0010] Determine the detection result of the model to be tested deployed in the machine learning platform according to the feature analysis result and the model metric evaluation result.

[0011] In a second aspect, an embodiment of the present invention provides a machine learning platform system for detecting a model, including:

[0012] An acquisition module, configured to acquire model detection data, where the model detection data includes prediction data, a prediction result obtained after the prediction data is input into the model to be tested, and evaluation data corresponding to the prediction data, and where the evaluation data is a result obtained by applying the prediction data to an actual business;

[0013] A feature analysis module, configured to perform feature analysis based on the prediction data and determine a feature analysis result when the prediction result and / or the evaluation data meet a preset model detection condition;

[0014] A model metric module, configured to determine a model metric evaluation result of the model to be tested based on the evaluation data;

[0015] An evaluation module, configured to determine a detection result of the model to be tested deployed in the machine learning platform according to the feature analysis result and the model metric evaluation result.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, including: a processor, a memory, and a program stored on the memory and executable on the processor, where when the program is executed by the processor, the steps of the method for model detection based on a machine learning platform as described in the first aspect above are implemented.

[0017] In an embodiment of the present invention, first, model detection data is obtained, where the model detection data includes prediction data, a prediction result obtained after the prediction data is input into the model to be detected, and evaluation data corresponding to the prediction data. The evaluation data is the result obtained when the prediction data is applied to the actual business. By introducing the evaluation data, the actual data support for model detection in the present invention is increased, and the effect of model detection is enhanced. When the prediction result and / or the evaluation data meet the preset model detection conditions, feature analysis is performed based on the prediction data to determine the feature analysis result. The feature analysis result obtained through feature analysis has higher accuracy than manual monitoring and reduces the influence of human factors on the analysis result. Further, based on the evaluation data, the model index evaluation result of the model to be detected is determined, and a model basis for judging model drift can be formed through the model index evaluation result. Finally, according to the feature analysis result and the model index evaluation result, the detection result of the model to be detected deployed in the machine learning platform is determined. The present invention obtains the feature analysis result and the model index evaluation result through the prediction data, the prediction result, and the evaluation data, and finally determines the detection result according to the feature analysis result and the model index evaluation result. Compared with the detection result obtained by manually monitoring the model, the accuracy of the final detection result is higher, and the final detection result is more persuasive. And when the preset model detection conditions are met, real-time model detection can be realized through the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0019] Figure 1 is a flowchart of a method for model detection based on a machine learning platform provided by an embodiment of the present invention;

[0020] Figure 2 is a flowchart of a simplified method for model detection based on a machine learning platform provided by an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of an interface for data return provided by an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of an interface for data statistics for model detection provided by an embodiment of the present invention;

[0023] Figure 5 is another schematic diagram of an interface for data statistics for model detection provided by an embodiment of the present invention;

[0024] Figure 6It is a schematic diagram of an interface for data distribution comparison in model detection provided by an embodiment of the present invention;

[0025] Figure 7 It is a schematic diagram of an interface for data anomaly statistics in model detection provided by an embodiment of the present invention;

[0026] Figure 8 It is a schematic diagram of an interface for model metric evaluation results in model detection provided by an embodiment of the present invention;

[0027] Figure 9 It is a schematic diagram of the structure of a machine learning platform system for detecting a model provided by an embodiment of the present invention;

[0028] Figure 10 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0029] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0030] Please refer to Figure 1 and Figure 2 , an embodiment of the present invention provides a method for model detection based on a machine learning platform, which is applied to a machine learning platform. The method includes:

[0031] Step 11: Obtain model detection data, where the model detection data includes prediction data, a prediction result obtained after the prediction data is input into the model to be tested, and evaluation data corresponding to the prediction data. Here, the evaluation data is the result obtained when the prediction data is applied to the actual business;

[0032] Step 12: When the prediction result and / or the evaluation data meet the preset model detection conditions, perform feature analysis based on the prediction data to determine the feature analysis result;

[0033] Step 13: Determine the model metric evaluation result of the model to be tested based on the evaluation data;

[0034] Step 14: Determine the detection result of the model to be tested deployed in the machine learning platform according to the feature analysis result and the model metric evaluation result.

[0035] In the embodiments of the present invention, first, model detection data is obtained. The model detection data includes prediction data, a prediction result obtained after the prediction data is input into the model to be detected, and evaluation data corresponding to the prediction data. The evaluation data is the result obtained when the prediction data is applied to the actual business. By introducing the evaluation data, the actual data support for model detection in the present invention is increased, and the effect of model detection is enhanced. When the prediction result and / or the evaluation data meet the preset model detection conditions, feature analysis is performed based on the prediction data to determine the feature analysis result. The feature analysis result obtained through feature analysis has higher accuracy than manual monitoring and reduces the influence of human factors on the analysis result. Further, based on the evaluation data, the model index evaluation result of the model to be detected is determined. The model index evaluation result can form a model basis for judging model drift. Finally, according to the feature analysis result and the model index evaluation result, the detection result of the model to be detected deployed in the machine learning platform is determined. The present invention obtains the feature analysis result and the model index evaluation result through the prediction data, the prediction result, and the evaluation data, and finally determines the detection result according to the feature analysis result and the model index evaluation result. The detection result has higher accuracy than that obtained by manual monitoring of the model, and the final detection result is more persuasive. And when the preset model detection conditions are met, real-time model detection can be realized through data.

[0036] In the embodiments of the present invention, optionally, the preset model detection conditions include at least one of the following:

[0037] The data volume of the prediction data or the evaluation data reaches the detection volume threshold, the prediction result is abnormal, and the difference value between the prediction result and the evaluation data is greater than the difference monitoring threshold.

[0038] Optionally, the data volume of the prediction data or the evaluation data reaching the detection volume threshold can be used as a model detection condition. Specifically: In the embodiments of the present invention or the machine learning platform in the present invention, the prediction data and / or evaluation data initiated by the business are automatically saved. When the data volume accumulates to the preset threshold, model detection is started, so as to obtain the feature analysis result and the model index evaluation result, and further obtain the detection result.

[0039] Further, optionally, the preset model detection conditions further include reaching the model detection cycle, and periodic detection results are obtained by configuring the detection cycle. The models launched in the machine learning platform need to be continuously and periodically evaluated and monitored, and each detection result is recorded. After comparative analysis, it can be used as an important basis for judging model drift. Therefore, periodic feature analysis results and model index evaluation results can be obtained by configuring the detection cycle, and further periodic detection results can be obtained.

[0040] Please refer to Figure 3, the present invention also provides an embodiment of data feedback. Specifically, when a user authorizes a machine learning platform to obtain their business data and after the data feedback is enabled in the embodiments of the present invention, the machine learning platform will receive prediction data and the results obtained by applying the prediction data to actual business, that is, evaluation data. The machine learning platform evaluates or trains the model through the prediction data, prediction results, and evaluation data. When configuring data feedback, since different prediction data may be stored in different locations, it is necessary to select prediction data sources and data tables, etc.; the running period can also be set, and the data feedback can be made to occur periodically by specifying the repetition period, repetition time, and crontab expression, so as to achieve the purpose of detecting or training the model within the specified period. And it is also necessary to further configure the running resources, such as CPU resources. When the data feedback is running, the running status, running results, and log information are recorded to achieve a better detection effect.

[0041] Please refer to Figure 4 , Figure 5 , Figure 6 and Figure 7 , in the embodiments of the present invention, optionally, the performing feature analysis based on the prediction data to determine the feature analysis result includes:

[0042] Obtain the training data of the model to be tested;

[0043] Perform feature analysis on the training data and the prediction data respectively to determine at least one of the feature distributions and statistical results corresponding to the training data and the prediction data, and the abnormal conditions of the prediction data;

[0044] Determine the feature analysis result according to at least one of the feature distribution, the statistical result, and the abnormal condition.

[0045] Specifically, after performing feature analysis on the training data and the prediction data respectively, determine at least one of the feature distribution of the training data, the statistical result of the training data, the feature distribution of the prediction data, the statistical result of the prediction data, and the abnormal conditions of the prediction data.

[0046] Among them, optionally, for example, the following algorithms can be used to analyze the data features of the training data and the prediction data respectively to determine their feature distributions and / or abnormal conditions: Population Stability Index (PSI) analysis and Kernel Density Estimation (KDE) analysis.

[0047] Optionally, please refer to Figure 7, according to the different types of feature variables, analyze the comparison of data metrics between the training data and the prediction data, and the statistical results include at least one of the following: maximum value Max, minimum value Min, mode Mode, value range Range, median Median, quartiles (Q1, Q3), etc.

[0048] Optionally, please refer to Figure 4 and Figure 5 , by performing valid value statistics, target column statistics, additional column statistics, and additional category statistics, determine the anomalies in the training data and the prediction data. In some embodiments, the situations where the training data or the prediction data has anomalies include, for example, column missing, data null value, field missing, type error, lack of valid values, lack of feature variables, outliers, and value out-of-bounds, etc. The embodiments of the present invention do not limit this.

[0049] The above display of feature distribution, statistical results, and anomalies can be in the form of a list, bar chart, line chart, etc., and different statistical or display methods are selected according to the characteristics of different data evaluation metrics.

[0050] Please refer to Figure 8 , in the embodiments of the present invention, optionally, determining the model metric evaluation result of the model to be tested based on the evaluation data includes:

[0051] Input the evaluation data and the prediction result into the machine learning platform to obtain the performance score of the model to be tested;

[0052] Based on the performance score, determine the model metric evaluation result of the model to be tested.

[0053] Optionally, the model metrics include at least one of the following: accuracy Accuracy, precision Precision, area under the receiver operating characteristic curve (Area under the ROC Curv, AUC).

[0054] In the embodiments of the present invention, when starting the monitoring and evaluation of the model, the model to be monitored needs to be selected first. The embodiments of the present invention support simultaneous monitoring and evaluation of multiple models. Please refer to Figure 8 , for example, the gradient boosting model, random forest multi-classification model, and decision tree model can be simultaneously monitored for their metrics to obtain their performance scores. The model metric evaluation result is determined according to the performance score, including the area under the receiver operating characteristic curve, accuracy, precision, and the harmonic mean F1 of recall rate, etc. Different model metrics are selected according to different model application scenarios. When configuring the data source and data table, the evaluation data can come from the new batch of data collected by the business party or the recycled evaluation data.

[0055] Further, determine the model metric evaluation result according to the performance score of the model to be tested. Specifically, taking the three models in Figure 8 as an example, the performance scores of the gradient progressive number model, the random forest multi-classification model, and the decision tree model are 0.81, 0.85, and 0.75 respectively. If the performance score is above 0.8, it means that the model is running stably. Then, it is easy to see that the model metric evaluation results of the gradient progressive number model and the random forest multi-classification model are no drift, and the decision tree model may have drift.

[0056] Further, the reason for model drift can be analyzed based on the feature analysis result and the model metric evaluation result. In the embodiments of the present invention, optionally, the determining the detection result of the model to be tested deployed in the machine learning platform according to the feature analysis result and the model metric evaluation result includes:

[0057] When the feature analysis result indicates that the prediction data is abnormal and the model metric evaluation result indicates that the performance score of the model to be tested decreases, determine that the detection result is that the model to be tested drifts, and there is a causal relationship between the drift of the model to be tested and the prediction data;

[0058] When the difference value between the feature distribution of the prediction data and the training data indicated by the feature analysis result is greater than the distribution threshold, and / or the difference value between the statistical results of the prediction data and the training data is greater than the statistical threshold, and the model metric evaluation result indicates that the performance score of the model to be tested decreases, determine that the detection result is that the model to be tested drifts, and there is a causal relationship between the drift of the model to be tested and the model to be tested.

[0059] Optionally, the data anomaly includes but is not limited to data null value, missing field, type error, missing valid value, or missing feature variable. When the feature analysis result of the model to be tested indicates data anomalies including but not limited to the above, and the performance score of the model to be tested decreases, then it is determined that the model to be tested drifts, and the drift reason is related to the prediction data. For example, when the feature analysis result of the model to be tested indicates a missing feature variable, and the performance score of the model to be tested decreases, for example, from 0.82 to 0.78, then the detection result is that the model to be tested drifts, and the reason for the drift is the missing feature variable.

[0060] Optionally, if the feature analysis result indicates that the difference value between the feature distribution of the predicted data and the feature distribution of the training data is greater than the distribution threshold. For example, if the PSI of the predicted data is compared with the PSI of the training data and the difference value between the feature distributions is greater than the distribution threshold, it indicates that there is a large change in the distribution. Moreover, when the model metric evaluation result indicates that the performance score of the model under test decreases, it is determined that the detection result is that the model under test has drifted, and the reason for the drift is that the ability of the model under test to process data has decreased.

[0061] Optionally, if the feature analysis result indicates that the difference value between the statistical result of the predicted data and the statistical result of the training data is greater than the statistical threshold. For example, if the maximum value, minimum value, mode, and value range of the predicted data, etc. are compared with the maximum value, minimum value, mode, and value range of the training data, etc., and the difference value between the statistical results is greater than the statistical threshold, it indicates that there is a large change in the statistical data result. Moreover, when the model metric evaluation result indicates that the performance score of the model under test decreases, it is determined that the detection result is that the model under test has drifted, and the reason for the drift is that the ability of the model under test to process data has decreased.

[0062] Optionally, if the feature analysis result indicates that the difference value between the feature distribution of the predicted data and the feature distribution of the training data is greater than the distribution threshold, and the difference value between the statistical result of the predicted data and the statistical result of the training data is greater than the statistical threshold, and the model metric evaluation result indicates that the performance score of the model under test decreases, it is determined that the detection result is that the model under test has drifted, and the reason for the drift is that the ability of the model under test to process data has decreased.

[0063] Furthermore, in an embodiment of the present invention, optionally, it further includes:

[0064] Select corresponding parameters according to the reason for the drift for quantization and normalization to obtain a normalized quantization index;

[0065] According to the business scenario of the model under test, select the weight coefficient corresponding to the parameter;

[0066] Based on the normalized quantization index and the weight coefficient corresponding to the normalized quantization index, determine the quantization index of the drift degree of the model under test.

[0067] Optionally, by quantizing and normalizing the parameters corresponding to the reason for the drift to obtain a normalized quantization index, and further based on the weight coefficient corresponding to the normalized quantization index determined according to the business scenario of the model under test, the quantization index of the drift degree of the model under test is determined. In this way, the quantization index can better support the detection result of the present invention from a data perspective. The quantization method makes the detection result more convincing and clearer.

[0068] Furthermore, the present invention also provides an embodiment for determining a quantitative index of the drift degree of the model to be tested, which can be quantified according to the parameters in the feature analysis result and the model index evaluation result, and the weights of the parameters. Enterprises can make the next decision based on this quantitative index.

[0069] Generally speaking, the indicators of the feature analysis result can include: valid value statistics, anomaly statistics, target column statistics, additional column statistics, additional categories, feature variable PSI index, feature variable KDE index, Min, Max, Median, Mode, Range, Q1, Q3, etc. The selection of indicators is based on the above-mentioned reasons for model drift; the model index evaluation can include: AUC, Accuracy, F1, Precision, etc. Of course, different model application scenarios result in different model index evaluation results.

[0070] Among them, the model drift score Drift Score can be calculated according to the following formula:

[0071] Drift Score = k i *c i = k 1 *c 1 + k 2 *c 2 +…+ k n *c n

[0072] Among them, k i represents the weight of the i-th indicator; c i represents the value of the i-th parameter corresponding to the drift reason after quantization.

[0073] Of course, different business scenarios have different requirements for indicator types and weight divisions. For example, in the bank credit risk control scenario, the bank needs to judge whether a user is a person who can obtain credit based on user data. Since the accuracy requirement for predicting creditworthy personnel is relatively high, the Precision index of the model has a greater weight than the Accuracy index or the AUC index. When the Drift Score exceeds the threshold, it indicates that model drift has occurred, and this model is no longer suitable for continued prediction. A new model needs to be selected based on the latest prediction data.

[0074] Furthermore, the embodiments of the present invention can simultaneously perform model drift detection on multiple models, so that when the online model drifts, a better-performing candidate model can be put back online. Among them, in the present invention, model drift detection can be provided by a machine learning platform. Specifically:

[0075] In the embodiments of the present invention, optionally, the model to be tested includes:

[0076] At least one of a main model and a plurality of candidate models, wherein the main model is the model currently used to provide prediction services for prediction data to be processed, and the candidate models have different parameters from the main model.

[0077] Optionally, different models to be tested (main model and candidate models) can be obtained based on different algorithms, or based on different parameters, or based on different data training sets.

[0078] Optionally, the embodiments of the present invention support deploying and going online multiple models, where the main model and the candidate models can be provided by the machine learning platform in the embodiments of the present invention, or can be provided by a third party.

[0079] Based on the above main model and candidate models, the subsequent iteration of the main model can be achieved through the following operations. Specifically: In the embodiments of the present invention, optionally, it further includes:

[0080] When the quantization index of the drift degree of the model to be tested is greater than the drift threshold, and there is a causal relationship between the drift of the model to be tested and the prediction data, optimize the prediction data according to the analysis result of the prediction data, and train the model to be tested according to the optimized prediction data;

[0081] When the quantization index of the drift degree of the model to be tested is greater than the drift threshold, and there is a causal relationship between the drift of the model to be tested and the model to be tested, update the training set according to the prediction data, and train the model to be tested based on the updated training set.

[0082] Optionally, when the quantization index of the drift degree of the model to be tested is greater than the drift threshold, and there is a causal relationship between the drift of the model to be tested and the prediction data, optimizing the prediction data according to the analysis result includes, but is not limited to, the following prediction data optimization methods. For example, if the data evaluation index shows data null values, field missing, or outliers, etc., delete the prediction data with data null values, field missing, or outliers, and re-enter the prediction data after deleting the null values, field missing, or outliers into the model to be tested to re-train the model to be tested and obtain an updated model to be detected. If the data evaluation index shows insufficient valid values, add more prediction data and re-enter it into the model to be tested to re-train the model to be tested and obtain an updated model to be detected. If the data evaluation index shows a lack of feature variables, re-enter the prediction data after adding new feature variables into the model to be tested for re-training. When the prediction data is abnormal, the prediction data needs to be reprocessed and used as updated prediction data to optimize and train the model according to the updated prediction data.

[0083] Optionally, when the quantization index of the drift degree of the model to be tested is greater than the drift threshold and there is a causal relationship between the drift of the model to be tested and the model to be tested, the training set can be updated according to the prediction data, and the model to be tested can be trained based on the updated training set. For example, if the detection result detected according to the above embodiment is that the model to be tested drifts, and the reason for the drift is that the ability of the model to be tested to process data decreases, and it is analyzed that the difference value between the feature distributions of the PSI of the prediction data and the PSI of the training data is greater than the distribution threshold, then the training set can be updated according to the parts with large differences in the feature distributions of the PSI of the prediction data and the PSI of the training data, and the model to be tested can be trained based on the updated training set, so as to obtain an optimized model to be tested, enhance the stability of the model to be tested, and maintain the performance of the model to be tested.

[0084] Optionally, as time goes by and data changes, the online model, that is, the main model, will drift, and its accuracy and performance will decline. Therefore, the fitness of the model can also be enhanced by replacing the main model. The replaced model can be a candidate model trained based on different algorithms, parameters or data. The candidate model that meets the online model requirements after being systematically evaluated by the machine learning platform in the embodiments of the present invention can better reflect the current business and data characteristics. The candidate model will replace the main model to continue to provide prediction tasks for the business system. When the candidate model goes online and becomes the main model, a model iteration process is completed. By iterating the main model, the accuracy of the online model can be maintained, a feasible processing mechanism can be formed, the negative impact brought by the drift of the main model can be reduced, and the stability of the model can be enhanced. The iteration period of the model can also be set so that the online model always remains the optimal model.

[0085] Please refer to Figure 9 , the present invention also provides a machine learning platform system 20 for detecting a model, including:

[0086] An acquisition module 21, configured to acquire model detection data, where the model detection data includes prediction data, a prediction result obtained after the prediction data is input into the model to be tested, and evaluation data corresponding to the prediction data, where the evaluation data is the result obtained when the prediction data is applied to the actual business;

[0087] A feature analysis module 22, configured to perform feature analysis based on the prediction data and determine a feature analysis result when the prediction result and / or the evaluation data meet a preset model detection condition;

[0088] A model index module 23, configured to determine a model index evaluation result of the model to be tested based on the evaluation data;

[0089] An evaluation module 24, configured to determine a detection result of a model under test deployed in the machine learning platform according to the feature analysis result and the model metric evaluation result.

[0090] Optionally, the preset model detection conditions include at least one of the following:

[0091] The data volume of the prediction data or the evaluation data reaches a detection volume threshold, the prediction result is abnormal, and the difference value between the prediction result and the evaluation data is greater than a difference monitoring threshold.

[0092] Optionally, the feature analysis module 22 includes:

[0093] A feature analysis sub-module, configured to obtain training data of the model under test;

[0094] Perform feature analysis on the training data and the prediction data respectively, and determine at least one of the feature distribution and statistical results corresponding to the training data and the prediction data, and the abnormal situation of the prediction data;

[0095] Determine the feature analysis result according to at least one of the feature distribution, the statistical result, and the abnormal situation.

[0096] Optionally, the model metric module 23 includes:

[0097] A model metric sub-module, configured to input the evaluation data and the prediction result into the machine learning platform to obtain a performance score of the model under test;

[0098] Determine a model metric evaluation result of the model under test based on the performance score.

[0099] Optionally, the evaluation module 24 includes:

[0100] An evaluation sub-module, configured to determine that the detection result is that the model under test drifts, and there is a causal relationship between the drift of the model under test and the prediction data when the feature analysis result indicates that the prediction data is abnormal and the model metric evaluation result indicates that the performance score of the model under test decreases;

[0101] When the feature analysis result indicates that the difference value between the feature distribution of the prediction data and the training data is greater than a distribution threshold, and / or the difference value between the statistical results of the prediction data and the training data is greater than a statistical threshold, and the model metric evaluation result indicates that the performance score of the model under test decreases, determine that the detection result is that the model under test drifts, and there is a causal relationship between the drift of the model under test and the model under test.

[0102] Optionally, the machine learning platform system 20 for detecting a model further includes:

[0103] A quantization module, configured to select corresponding parameters for quantization and normalization according to the drift reason, and obtain a normalized quantization index;

[0104] Select a weight coefficient corresponding to the parameter according to the business scenario of the model to be tested;

[0105] Based on the normalized quantization index and the weight coefficient corresponding to the normalized quantization index, determine a quantization index for the drift degree of the model to be tested.

[0106] Optionally, the model to be tested includes:

[0107] At least one of a main model and a plurality of candidate models, where the main model is a model currently used to provide prediction services for the service data to be processed, and the parameters of the candidate models are different from those of the main model.

[0108] Optionally, the machine learning platform system 20 for detecting a model further includes:

[0109] An optimization module, configured to, if the quantization index of the drift degree of the model to be tested is greater than a drift threshold,

[0110] If the drift reason of the model to be tested is data, optimize the prediction data according to the reason for the data anomaly, and re-enter the optimized prediction data into the model to be tested for re-prediction.

[0111] The machine learning platform system 20 for detecting a model provided by an embodiment of the present invention can implement Figures 1 to 8 each process implemented by the method embodiment, and achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0112] An embodiment of the present invention provides an electronic device 30. Refer to Figure 10 as shown, Figure 10 is a schematic block diagram of the electronic device 30 according to an embodiment of the present invention, including a processor 31, a memory 32, and a program or instruction stored in the memory 32 and executable on the processor 31. When the program or instruction is executed by the processor, the steps in any one of the model detection methods based on the machine learning platform of the present invention are implemented.

[0113] An embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the embodiment of the model drift detection method as described above is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be elaborated here.

[0114] A computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage, or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0115] It should be noted that in this article, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising such element.

[0116] The serial numbers of the above-described embodiments of the present invention are for description purposes only and do not represent the superiority or inferiority of the embodiments.

[0117] From the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a service classification device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present application.

[0118] The above is only the preferred embodiment of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for model detection based on a machine learning platform, characterized in that, it includes: Obtain model detection data, where the model detection data includes prediction data, a prediction result obtained after inputting the prediction data into the model to be tested, and evaluation data corresponding to the prediction data, where the evaluation data is the result obtained by applying the prediction data to the actual business; When the prediction result and / or the evaluation data meet the preset model detection conditions, perform feature analysis based on the prediction data to determine the feature analysis result; Based on the evaluation data, determine the model index evaluation result of the model to be tested; According to the feature analysis result and the model index evaluation result, determine the detection result of the model to be tested deployed in the machine learning platform.

2. The method according to claim 1, characterized in that, the preset model detection conditions include at least one of the following: The data volume of the prediction data or the evaluation data reaches the detection volume threshold, the prediction result is abnormal, and the difference value between the prediction result and the evaluation data is greater than the difference monitoring threshold.

3. The method according to claim 1, characterized in that, the performing feature analysis based on the prediction data to determine the feature analysis result includes: Obtain the training data of the model to be tested; Perform feature analysis on the training data and the prediction data respectively to determine at least one of the feature distribution and statistical results corresponding to the training data and the prediction data, and the abnormal situation of the prediction data; Determine the feature analysis result according to at least one of the feature distribution, the statistical result, and the abnormal situation.

4. The method according to any one of claims 1-3, characterized in that, the determining the model index evaluation result of the model to be tested based on the evaluation data includes: Input the evaluation data and the prediction result into the machine learning platform to obtain the performance score of the model to be tested; Based on the performance score, determine the model index evaluation result of the model to be tested.

5. The method according to any one of claims 1-4, characterized in that, the determining the detection result of the model to be tested deployed in the machine learning platform according to the feature analysis result and the model index evaluation result includes: When the feature analysis result indicates that the prediction data is abnormal, and the model index evaluation result indicates that the performance score of the model to be tested decreases, determine that the detection result is that the model to be tested drifts, and there is a causal relationship between the drift of the model to be tested and the prediction data; When the feature analysis result indicates that the difference value between the feature distribution of the prediction data and the training data is greater than the distribution threshold, and / or the difference value between the statistical results of the prediction data and the training data is greater than the statistical threshold, and the model index evaluation result indicates that the performance score of the model to be tested decreases, determine that the detection result is that the model to be tested drifts, and there is a causal relationship between the drift of the model to be tested and the model to be tested.

6. The method according to claim 5, characterized in that, it further includes: Select corresponding parameters according to the drift cause for quantification and normalization to obtain the normalized quantification index; Select the weight coefficient corresponding to the parameter according to the business scenario of the model to be tested; Based on the normalized quantification index and the weight coefficient corresponding to the normalized quantification index, determine the quantification index of the drift degree of the model to be tested.

7. The method according to claim 6, wherein, the model to be tested includes: at least one of a main model and several candidate models, wherein the main model is the model currently used to provide prediction services for the prediction data to be processed, and the candidate models have different parameters from the main model.

8. The method according to claim 5, wherein, further includes: In the case where the quantification index of the drift degree of the model to be tested is greater than the drift threshold and there is a causal relationship between the drift of the model to be tested and the prediction data, optimize the prediction data according to the analysis result of the prediction data, and train the model to be tested according to the optimized prediction data; In the case where the quantification index of the drift degree of the model to be tested is greater than the drift threshold and there is a causal relationship between the drift of the model to be tested and the model to be tested, update the training set according to the prediction data, and train the model to be tested based on the updated training set.

9. A machine learning platform system for detecting a model, wherein, includes: An acquisition module for acquiring model detection data, where the model detection data includes prediction data, a prediction result obtained after the prediction data is input into the model to be tested, and evaluation data corresponding to the prediction result; A feature analysis module for performing feature analysis based on the prediction data in the case where the prediction result and the evaluation data meet the preset model detection conditions to determine a feature analysis result; A model index module for determining an evaluation result of the model index of the model to be tested based on the evaluation data; An evaluation module for determining a detection result of the model to be tested deployed in the machine learning platform according to the feature analysis result and the model index evaluation result.

10. An electronic device, wherein, includes: A processor, a memory, and a program stored on the memory and executable on the processor, and when the program is executed by the processor, the steps of the method for model detection based on a machine learning platform according to any one of claims 1 to 9 are implemented.

Images