Model score offset detection method and device, equipment and storage medium

Abstract

The invention discloses a model score offset detection method, device and equipment and a storage medium, samples of a first time period sample set between adjacent preset sample nodes are filled by adopting an interpolation function to obtain a corresponding variable filling value, a corresponding number of virtual samples are generated, a second time period sample set is filled with the virtual samples, and the corresponding variable filling value is obtained. Obtaining a second time period new sample set, sorting the sample variables of the first time period sample set and the sample variables of the second time period new sample set according to a preset variable contribution degree sorting mode, and obtaining a first sample variable set and a second sample new variable set in a corresponding sequence, and replacing the variables of the second sample new variable set with the variables of the first sample variable set one by one according to a corresponding sequence, calculating an individual variable error, and determining a model score offset degree according to the variable error. According to the method, when the model is applied to samples in different time periods, when the model score has large offset, how to determine the variable with the large error of the single variable and how to correspondingly analyze the reason of the model score offset are realized.

Description

Technical Field

[0001] The present application relates to the technical field of mathematical model processing, and in particular to a model score deviation detection method, device, equipment and storage medium. Background Art

[0002] Model score drift attribution refers to the process of analyzing the causes of deviations between model prediction scores and actual results in machine learning models. In credit scoring models or other predictive models, model scores usually represent the model's confidence or probability estimate that a sample belongs to a certain class or a certain result. When model scores change over different time periods or on different data sets, drift attribution analysis may be required.

[0003] Attribution shift is the difference between the model's prediction and its expected output. This difference may be caused by a variety of factors, including but not limited to changes in data distribution, limitations of the model itself, insufficient feature engineering, or overfitting of the model on new data. In practical applications, when the model's prediction results deviate significantly from expectations, we need to perform attribution shift analysis to determine the cause of this deviation. This usually involves the following steps: Data distribution check: Check whether the distribution of training data and test data is significantly different, which may involve statistical tests, visualization methods, etc. Model performance evaluation: Use various metrics (such as accuracy, recall, F1 score, etc.) to evaluate the performance of the model. Feature importance analysis: Determine which features have the greatest impact on the model's prediction results, which can be achieved through methods such as feature importance ranking, SHAP value analysis, etc. Model diagnosis: Use model diagnostic tools and techniques to identify possible problems with the model, such as overfitting, underfitting, etc. Model update: Based on the results of the attribution analysis, the model may need to be retrained or adjusted to improve its performance on new data. In some cases, drift attribution may also involve the problem of domain adaptation (DA), that is, how to apply a model trained in one domain (source domain) to a different domain (target domain). This usually involves techniques such as generative adversarial networks (GANs) to reduce the difference in feature distribution between the source and target domains. In addition, drift attribution may also involve the real-time data distribution shift of the model in the production environment, which may cause a corresponding shift in the feature attribution value.

[0004] When the current model is applied to samples in different time periods, the expected score is stable. If a large deviation occurs, the cause of the deviation needs to be analyzed. However, the current mainstream machine learning algorithms themselves have a weak explanation of the cause of the deviation, and each algorithm has different principles. For example, the tree model and logistic regression have different definitions of feature importance, and the deviation attribution needs to be processed for different models, which leads to technical problems such as low calculation efficiency of mathematical models.

[0005] In view of this, the existing technology needs to be improved and enhanced. Summary of the invention

[0006] The main purpose of the present invention is to provide a model score offset detection method, device, equipment and storage medium, aiming to solve the current mainstream machine learning algorithms, which have weak explanations of the causes of offsets and different principles of each algorithm. For example, tree models and logistic regression have different definitions of feature importance, and offset attribution processing needs to be performed for different models, resulting in technical problems in low efficiency in mathematical model calculations.

[0007] To achieve the above object, the present invention provides a model score deviation detection method, which comprises:

[0008] Filling the samples of the first time period sample set between adjacent preset sample nodes with an interpolation function to obtain corresponding variable filling values, and generating a corresponding number of virtual samples according to the variable filling values;

[0009] Filling the virtual sample into the sample set of the second period to obtain a new sample set of the second period;

[0010] Sort the sample variables of the sample set of the first period and the sample variables of the new sample set of the second period according to the preset variable contribution sorting method, and obtain the first sample variable set and the second sample new variable set in the corresponding order;

[0011] The variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in the corresponding order, the individual variable errors are calculated, and the degree of model score deviation is determined according to the variable errors.

[0012] The model score deviation detection method, wherein the samples of the first time period sample set between adjacent preset sample nodes are filled with an interpolation function to obtain corresponding variable filling values, and a corresponding number of virtual samples are generated according to the variable filling values, further comprising:

[0013] Sorting the samples of the sample set in the first time period according to the preset model sorting method, and calculating the corresponding quantiles;

[0014] Sample nodes are calculated according to the quantiles, and correspond one to one with all samples of the sample set in the first time period.

[0015] The model score deviation detection method, wherein the samples of the first time period sample set between adjacent preset sample nodes are filled with an interpolation function to obtain corresponding variable filling values, and a corresponding number of virtual samples are generated according to the variable filling values, specifically includes:

[0016] Using a cubic spline interpolation function to sequentially fill in the sample model points of the first time period sample set adjacent to the sample nodes;

[0017] Obtain variable filling values ​​corresponding to the number of variables after filling the sample set of the first period;

[0018] Generate a corresponding number of the virtual samples according to the variable filling value.

[0019] The model score deviation detection method, wherein the variables of the first period sample set and the variables of the second period new sample set are sorted according to a preset variable contribution sorting method, and the first sample variable set and the second sample new variable set in the corresponding order are obtained, specifically comprising:

[0020] Sorting the sample variables of the first time period sample set according to the preset variable contribution ranking method to obtain the first sample variable set in a corresponding order;

[0021] Sorting the sample variables of the new sample set in the second time period according to the preset variable contribution ranking method to obtain the second sample variable set in the corresponding order;

[0022] The samples of the sample set in the first time period and the samples of the new sample set in the second time period are sorted according to a preset model sorting method.

[0023] The model score deviation detection method, wherein the variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in a corresponding order, the individual variable errors are calculated, and the degree of model score deviation is determined according to the variable errors, specifically comprising:

[0024] Matching the samples of the sample set of the first time period sorted according to the preset model sorting method with the samples of the new sample set of the second time period one by one;

[0025] The variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in the corresponding order, the individual variable errors are calculated, and the degree of model score deviation is determined according to the variable errors.

[0026] The model score shift detection method, wherein the variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in a corresponding order, and the individual variable errors are calculated, specifically includes:

[0027] Replace the variables of the second sample new variable set one by one in the corresponding order with the variables of the first sample variable set;

[0028] Calculate the individual variable error each time after the variables of the first sample variable set replace the variables of the second sample new variable set;

[0029] The errors of all the individual variables are summed to obtain the model average error.

[0030] The model score deviation detection method, wherein the preset variable contribution degree sorting method is variable contribution degree ascending order or variable contribution degree descending order;

[0031] The preset model score sorting method is model score ascending order or model score descending order.

[0032] Furthermore, to achieve the above object, the present invention also provides a model score deviation detection device, which comprises:

[0033] An interpolation filling module, used to fill the samples of the first time period sample set between adjacent preset sample nodes using an interpolation function to obtain corresponding variable filling values, and generate a corresponding number of virtual samples according to the variable filling values;

[0034] A new sample generation module, used for filling the virtual sample into the second period sample set to obtain a new sample set for the second period;

[0035] A sample variable module, used to sort the sample variables of the sample set of the first period and the sample variables of the new sample set of the second period according to a preset variable contribution sorting method, and obtain the first sample variable set and the second sample new variable set in corresponding order;

[0036] The variable replacement module is used to replace the variables of the second sample new variable set one by one with the variables of the first sample variable set in the corresponding order, calculate the error of the individual variable, and determine the degree of model score deviation according to the variable error.

[0037] Furthermore, to achieve the above object, the present invention also provides a model score deviation detection device, the device comprising a processor and a memory:

[0038] The memory is used to store program code and transmit the program code to the processor;

[0039] The processor is used to execute the above-mentioned model score deviation detection method according to the instructions in the program code.

[0040] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store program code, and the program code is used to execute the above-mentioned model score deviation detection method.

[0041] Beneficial effects: The present application provides a method, device, equipment and storage medium for detecting model score deviation. The interpolation function is first used to fill the samples of the first time period sample set between adjacent preset sample nodes to obtain the corresponding variable filling value, and a corresponding number of virtual samples are generated. The virtual samples are filled into the second time period sample set to obtain the second time period new sample set. The sample variables of the first time period sample set and the sample variables of the second time period new sample set are sorted according to the preset variable contribution ranking method, and the first sample variable set and the second sample new variable set of corresponding order are obtained. The variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in the corresponding order, and the individual variable error is calculated. The degree of model score deviation is determined according to the variable error. The model is applied to samples of different time periods. When the model score deviates greatly, how to determine the variable with the larger individual variable error and analyze the cause of the model score deviation accordingly. At the same time, the present application uses a cubic spline interpolation function to sequentially fill the sample model scores of the first time period sample set adjacent to the sample nodes, so that the distribution of virtual samples is smoother. . BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0043] Figure 1 This is a method flow chart of the model score deviation detection method in an embodiment of the present application;

[0044] Figure 2 This is a schematic diagram of the structure of a model scoring deviation detection device in an embodiment of the present application;

[0045] Figure 3 Schematic diagram of the structure of the model scoring deviation detection device in the embodiment of the present application. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0047] The present application designs a method for detecting model score offset, which first uses an interpolation function to fill in samples of a first time period sample set between adjacent preset sample nodes to obtain corresponding variable filling values, and generates a corresponding number of virtual samples, which are then filled into a second time period sample set to obtain a new sample set for the second time period, and the sample variables of the first time period sample set and the sample variables of the second time period new sample set are sorted according to the preset variable contribution sorting method, and the first sample variable set and the second sample new variable set of corresponding order are obtained, and the variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in the corresponding order, and the error of the individual variable is calculated, which solves the problem that the current model is applied to samples of different time periods. If the model score has a large offset, it is necessary to analyze the cause of the offset. However, the current mainstream machine learning algorithms themselves have a weak explanation of the cause of the offset, and each algorithm has different principles. For example, tree models and logistic regression have different definitions of feature importance, and it is necessary to perform offset attribution processing for different models, which leads to technical problems in the low efficiency of mathematical model calculations. For ease of understanding, please refer to Figure 1 , Figure 1 : is a flow chart of the method for detecting model score deviation in the embodiment of the present application. Figure 1 As shown, specifically:

[0048] S101, using an interpolation function to fill samples of a first time period sample set between adjacent preset sample nodes to obtain corresponding variable filling values, and generating a corresponding number of virtual samples according to the variable filling values;

[0049] It should be noted that the interpolation function is used to fill the samples of the first time period sample set between adjacent preset sample nodes to obtain corresponding variable filling values ​​and generate a corresponding number of virtual samples, which also includes:

[0050] Sorting the samples of the sample set in the first time period according to the preset model sorting method, and calculating the corresponding quantiles;

[0051] Sample nodes are calculated according to the quantiles, and correspond one to one with all samples of the sample set in the first time period.

[0052] Meanwhile, in one embodiment, an interpolation function is used to fill samples of the first time period sample set between adjacent preset sample nodes to obtain corresponding variable filling values ​​and generate a corresponding number of virtual samples, specifically including:

[0053] Using a cubic spline interpolation function to sequentially fill in the sample model points of the first time period sample set adjacent to the sample nodes;

[0054] Obtain variable filling values ​​corresponding to the number of variables after filling the sample set of the first period;

[0055] Generate a corresponding number of the virtual samples according to the variable filling value.

[0056] The specific applications are as follows:

[0057] For example, there is a total error TME in the average scores of sample sets A and B from two time periods.

[0058] Usually the number of A and B is different. Assume that the number of samples in A is n more than that in B, and the number of variables is m. Perform the following steps:

[0059] 1. Sort the model of A into n quantiles, with a total of n+1 nodes, corresponding to the samples in A

[0060] 2. Assuming the coordinate system of model scores and variable values, the horizontal axis is the model score of the above sample, and the vertical axis is the corresponding var1 value. Using the cubic spline interpolation function, the random positions in the two samples can be filled with 1 var1 value;

[0061] var' x =cubic_spline(var x-1 ,var x ), where var x-1 , var x It is a variable value in the interval [,-1], and is the abbreviation of cubic_spline cubic spline interpolation function.

[0062] 3. Repeat step 2 until all m variables are filled.

[0063] The aforementioned cubic spline interpolation function is used to create a smooth curve that passes through a series of data points. This interpolation method uses a series of cubic polynomial functions, each defined in a subinterval of the data points. The characteristics of cubic spline interpolation are: Smoothness: At each data point (node), not only do the function values ​​match, but the first and second derivatives of the function are also continuous. Local control: Each polynomial segment only affects its adjacent interval, so small modifications to the curve will not affect the entire curve. Flexibility: It can adapt to local changes in the data without being too rigid like linear interpolation. Boundary conditions: Different boundary conditions can be specified, such as natural boundary conditions (the second-order derivative at both ends is zero) or fixed boundary conditions (specify the slope of the two end points).

[0064] In practical applications, cubic spline interpolation can be used for function approximation in scientific computing and engineering, curve design in computer graphics, and data smoothing and missing value filling in statistics.

[0065] S102, filling the virtual sample into the sample set of the second period to obtain a new sample set of the second period;

[0066] After using an interpolation function to fill samples of the first time period sample set between adjacent preset sample nodes to obtain corresponding variable filling values ​​and generate a corresponding number of virtual samples, the virtual samples can be filled into the second time period sample set to obtain a new second time period sample set.

[0067] In the above embodiment, for a1, ..., an, the padding values ​​on the left of their respective variables are taken to generate a new sample, that is, n virtual samples are generated. The n virtual samples are filled into B, and the number of samples of A and new_B is the same.

[0068] It should be noted that, in one embodiment, for discrete variables, the virtual samples use the values ​​of a1 to an themselves.

[0069] S103, sorting the sample variables of the first period sample set and the sample variables of the second period new sample set according to a preset variable contribution sorting method, and obtaining the first sample variable set and the second sample new variable set in corresponding order;

[0070] It can be understood that sorting the sample variables of the first time period sample set and the sample variables of the second time period new sample set according to the preset variable contribution sorting method, and obtaining the first sample variable set and the second sample new variable set in corresponding order specifically includes:

[0071] Sorting the sample variables of the first time period sample set according to the preset variable contribution ranking method to obtain the first sample variable set in a corresponding order;

[0072] Sorting the sample variables of the new sample set in the second time period according to the preset variable contribution ranking method to obtain the second sample variable set in the corresponding order;

[0073] The samples of the sample set in the first time period and the samples of the new sample set in the second time period are sorted according to a preset model sorting method.

[0074] It can be understood that the preset variable contribution ranking method is variable contribution ascending order or variable contribution descending order, and the preset model score ranking method is model score ascending order or model score descending order.

[0075] In a specific application embodiment, for example:

[0076] Sort the m model variables from high to low according to their contribution, and get var1~varm. In the A sample and the B sample, the number of variables is m, and var1~varm is the m model variables;

[0077] Sort A and new_B from high to low according to the model scores. new_B is the new sample after the aforementioned B sample is filled.

[0078] S104. Replace the variables of the second sample new variable set with the variables of the first sample variable set one by one in the corresponding order, calculate the individual variable errors, and determine the degree of model score deviation according to the variable errors.

[0079] In this embodiment, the variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in a corresponding order, and the individual variable errors are calculated, which may be specifically:

[0080] Matching the samples of the sample set of the first time period sorted according to the preset model sorting method with the samples of the new sample set of the second time period one by one;

[0081] The variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in the corresponding order, and the individual variable errors are calculated.

[0082] In one embodiment, 1. Replace the var2-varm values ​​of each sample in new_B with the corresponding var2-varm values ​​in A. Calculate the average error var1_ME between A and new_B'. new_B' is the sample of the result of new_B being replaced, and var1_ME represents the error caused by the difference in var1 alone.

[0083] 2. Replace the values ​​of var3~varm of each sample in new_B with the corresponding values ​​in A. Calculate the average error var2_ME between A and new_B'.

[0084] 3. Repeat this step until the single varm is replaced. At this time, A and new_B' are exactly the same without error, that is, mdl_ME is split into:

[0085] Where mdl_ME is the average error between A and new_B, is the sum of the mean errors of all individual variables.

[0086] It should be noted that the aforementioned model error changes after replacing some variables (var2 to varm) in the sample set new_B with the corresponding variables in the sample set A. Here, var1_ME refers to the error caused by the difference in the first variable (var1) after the replacement.

[0087] Here are the detailed steps of this process:

[0088] Prepare data: Make sure both sample sets A and new_B contain the same number of variables m, and their var2 to varm are comparable.

[0089] Replace variables: For each sample in new_B, replace the values ​​of var2 to varm with the values ​​of var2 to varm in the same order in sample set A. In this way, new_B becomes new_B'.

[0090] Calculate var1_ME: In new_B', only the value of var1 is different from the value in A. Use the model to predict var1 for new_B' and calculate the error between the predicted value and the actual value. This error is var1_ME, which represents the error caused by the difference in var1 alone.

[0091] Average error: Calculate the average of var1_ME of all new_B' samples to get the average error var1_ME.

[0092] This process can understand the impact of var1 on the model prediction error when other variables remain unchanged. If the model is very sensitive to var1, then var1_ME may be relatively large. On the contrary, if the model is not sensitive to changes in var1, then var1_ME may be small.

[0093] Note that this procedure assumes that the substitution of var2 to varm does not affect the prediction of var1. In real situations, there may be interactions between variables, so this substitution may indirectly affect the prediction of var1. In addition, this procedure also assumes that A and new_B differ enough in var1 to be able to observe the effect of the substitution.

[0094] The attribution of the overall deviation between sample A in the first period and sample B in the second period is expressed as:

[0095] Among them, fill_ME is the filling sample, is the sum of the mean errors of all individual variables.

[0096] In this embodiment, the average error mdl_ME between A and new_B is calculated, then fill_ME=TME-mdl_ME, fill_ME is called the filling error, which means that the error fill_ME comes from the unequal number of A and B and the randomness of filling.

[0097] There are two sample sets A and B, and a new sample set new_B. If you want to calculate the mean error of the model (mdl_ME), and then calculate the filling error (fill_ME), this filling error is caused by the unequal number of samples and the randomness of the filling. Here are the general steps of this process:

[0098] Defining Error: First, we need to define what model error (mdl_ME) is. This is usually the difference between the model's predicted value and the actual value. In statistics, this can be calculated using various error metrics, such as mean squared error (MSE) or mean absolute error (MAE).

[0099] Calculate mdl_ME: For sample sets A and new_B, you can use the corresponding statistical method to calculate the model error. This usually involves comparing the model's predicted values ​​with the actual values ​​and calculating a measure of the error.

[0100] Definition of filling error: Filling error (fill_ME) is the additional error caused by unequal sample sizes and randomness of filling. This concept assumes that if the sample sizes are equal, the model error (mdl_ME) will be reduced.

[0101] Calculate the filling error: The filling error can be calculated by subtracting the model error (mdl_ME) from the total error (TME). The total error (TME) here refers to the ideal error that the model can achieve without any sample size inequality and filling randomness.

[0102] Analysis of the reason: The calculation of the filling error can help analyze the impact of sample size inequality and filling randomness on model performance. If fill_ME is large, this may indicate that sample size inequality and filling randomness have a significant impact on the model's predictive performance.

[0103] In practical applications, this process may require more detailed data and complex statistical analysis. For example, it may be necessary to use a machine learning model to predict the value of new_B and calculate the error between the predicted value and the actual value. Then, TME can be estimated through experiments or simulations, and mdl_ME can be subtracted from it to get fill_ME.

[0104] Rating drift attribution is a complex process involving multiple factors and methods, and it has different implementation methods and purposes in different application scenarios. The model rating drift detection of this embodiment can be applied to the following application scenarios:

[0105] Bias correction in competition scoring: This method uses statistical principles, such as the normal distribution assumption, to adjust the scoring standards of each judge to ensure that all judges' scores are compared under the same standards. This usually involves calculating the mean score and standard deviation of each judge, and then transforming the raw scores to eliminate differences between raters.

[0106] Performance attribution in fund evaluation systems: In the financial field, performance attribution analysis helps investors understand how fund managers' investment decisions affect fund performance. Timing and stock selection ability attribution analyzes whether fund managers are able to buy or sell stocks at the right time and whether they have selected stocks that perform well. Style attribution analyzes whether the fund's investment style matches market trends.

[0107] Rater drift in subjective question scoring: In educational and psychological assessments, rater drift may affect the consistency and reliability of scoring. Statistical tools such as the multifaceted Rasch model can be used to detect and quantify rater drift so that measures can be taken to reduce scoring errors and improve scoring accuracy.

[0108] The aforementioned Multi-Facet Rasch Model (MFRM) is able to assess individual ability, the difficulty of questions, and the consistency and bias of raters (judges) under more complex test conditions, such as structured interviews. This model is particularly suitable for dealing with complex measurement problems involving multiple factors (facets), such as assessing student ability, question difficulty, and the rigor of raters in educational tests.

[0109] This embodiment is particularly applicable to risk control models in credit scenarios, response models in marketing scenarios, fault detection models in production scenarios, etc.

[0110] It can be further understood that the mathematical model mentioned in this embodiment can be a model such as logistic regression, LGBM, and random forest.

[0111] Among them, Logistic Regression: Logistic regression is a linear model widely used in binary classification problems. It uses a logical function (usually a Sigmoid function) to map the results of linear regression to between 0 and 1 to predict the probability of an event. The output value of logistic regression can be interpreted as the probability of belonging to a certain class. Its advantages are that the model is simple, easy to understand and implement, and has high computational efficiency. The disadvantages are that it is easily affected by outliers and assumes that the data is linear, which may not always be true in the real world. LGBM (Light Gradient Boosting Machine) machine learning algorithm: LightGBM is an algorithm based on gradient boosting decision trees (GBDT), which improves the performance of the model by building multiple decision trees. It uses a tree-based learning algorithm and optimizes memory usage and computing speed, which is particularly suitable for processing large-scale data. LightGBM supports binary, multi-classification, and regression problems, and supports parallel and GPU acceleration. Its advantages include high efficiency when processing large data sets and the ability to handle sparse data. Disadvantages may include the complexity of the model and dependence on parameter tuning. Understandably, the LGBM machine learning algorithm has: Faster training speed and higher efficiency: LightGBM uses a histogram-based algorithm to speed up training and reduce memory usage by segmenting continuous eigenvalues ​​into discrete bins. Lower memory usage: Compared with other pre-sorted-based algorithms, LightGBM reduces memory usage. Better accuracy: LightGBM has shown excellent accuracy in many machine learning tasks. Support for parallel, distributed, and GPU learning: LightGBM supports feature parallelism, data parallelism, and voting parallelism, and can process large-scale data. Sparse optimization: LightGBM optimizes sparse data and improves the efficiency of processing sparse data.

[0112] The implementation features of LightGBM include: Histogram-based decision tree algorithm: LightGBM uses a histogram algorithm instead of a pre-sorting algorithm, which can reduce the amount of calculation and improve efficiency. Leaf-wise leaf growth strategy: LightGBM uses the best-first leaf growth strategy, and each time finds the leaf with the largest split gain from all current leaves for splitting, which can reduce errors and improve accuracy. Histogram difference acceleration: When constructing a histogram, LightGBM can obtain it by subtracting the histogram of a leaf from the histogram of its brother, which can reduce communication and increase speed. It should be noted that when using LightGBM to process unbalanced data sets, the following methods can be adopted: Adjust category weights: Adjust the weights of positive and negative samples by setting the scale_pos_weight parameter, so that the model pays more attention to minority classes during training. For example, if the ratio of majority class to minority class is 9:1, you can set scale_pos_weight=9. Resampling: Use oversampling methods such as SMOTE (Synthetic Minority Over-sampling Technique) to increase the number of samples in the minority class, or use undersampling methods to reduce the number of samples in the majority class. Resampling can be done in the data preprocessing stage, or it can be achieved by setting the weight parameter during the training process of LightGBM. Use FocalLoss: Focal Loss is a loss function designed specifically to solve the problem of class imbalance. It focuses on samples that are difficult to classify by reducing the weight of easy-to-classify samples and increasing the weight of difficult-to-classify samples. Focal Loss can be used to enhance LightGBM's ability to handle imbalanced data. Adjust evaluation indicators: When evaluating the model, not only accuracy is used as an evaluation indicator, but also more comprehensive indicators such as precision, recall, F1 score, and area under the ROC curve (AUC) can be used to evaluate model performance. Try different LightGBM parameters: For example, adjust parameters such as num_leaves, min_data_in_leaf, max_depth, etc. to control the complexity and overfitting of the model. Use synthetic minority samples: SMOTE is a popular oversampling technique that creates new synthetic samples by interpolating between minority class samples. Ensemble methods: Ensemble methods that use multiple models, such as Bagging or Boosting, can help improve the model's predictive performance on minority classes. Cost-sensitive learning: Different misclassifications are assigned different penalty weights, making the model pay more attention to misclassifications of minority classes. Use custom evaluation metrics: LightGBM allows users to define their own evaluation metrics, which can be used to better evaluate the performance of the model on unbalanced datasets.Data post-processing: Calibrate the model’s predictions, such as by adjusting the threshold to improve the predictions for the minority class.

[0113] Random Forest: Random Forest is an ensemble learning method that improves the accuracy and stability of the model by building multiple decision trees and aggregating their results. Each tree uses a randomly selected subset of features during training, which can increase the generalization ability of the model and reduce the risk of overfitting. Random Forest can be used for classification and regression problems, and usually does not require much parameter tuning. Its advantages are that the stability and accuracy of the model are usually high and it is not sensitive to feature scaling. The disadvantages are that the model may become complex, require more computing resources, and the model is not as interpretable as a single tree model.

[0114] In this embodiment, the offset analysis can specifically determine which variables have the greatest impact on the model prediction error by comparing the error contribution of each variable (var1_ME, var2_ME, ..., varm_ME). Model optimization: After identifying the variables that contribute more to the error, you can consider optimizing the data collection, feature engineering, or model input of these variables. Decision making: Variable offset analysis can help business decision makers understand which variables are most critical in the decision-making process, so as to make more targeted strategy adjustments. Data quality improvement: If the error contribution of a variable is large, it may be due to data quality issues, such as missing values, outliers, etc., which can guide data cleaning and preprocessing.

[0115] This process requires careful design and execution to ensure consistency and fairness in the replacement process. In addition, this analysis involves complex data processing and statistical analysis.

[0116] The model score offset detection method in this embodiment first uses an interpolation function to fill the samples of the first time period sample set between adjacent preset sample nodes to obtain corresponding variable filling values, and generates a corresponding number of virtual samples, fills the virtual samples into the second time period sample set, obtains a new sample set for the second time period, sorts the sample variables of the first time period sample set and the sample variables of the second time period new sample set according to the preset variable contribution sorting method, and obtains the first sample variable set and the second sample new variable set in the corresponding order, replaces the variables of the first sample variable set with the variables of the second sample new variable set one by one in the corresponding order, and calculates the error of the individual variable. This is because the current model is applied to samples of different time periods. If the model score has a large offset, it is necessary to determine the variable with a large individual variable error to analyze the cause of the offset of the mathematical model score. However, the current mainstream machine learning algorithm itself has a weak explanation of the cause of the offset, and each algorithm has a different principle. For example, the tree model and logistic regression have different definitions of feature importance, and it is necessary to perform offset attribution processing for different models, which leads to a technical problem of low efficiency in mathematical model calculation.

[0117] At the same time, the present application adopts a cubic spline interpolation function to sequentially fill in the sample model points of the first time period sample set of adjacent sample nodes, so that the distribution of virtual samples is smoother.

[0118] Furthermore, it is possible to quickly locate the main cause of score deviation, determine whether it is internal or external, whether the variables are stable, and whether the model needs to be iterated, thus solving the problem of no universal model score deviation analysis method at one time and not being limited to model methods.

[0119] See also Figure 2 , Figure 2 Schematic diagram of the structure of the model scoring deviation detection device in the embodiment of the present application. Figure 2 As shown, specifically:

[0120] The interpolation filling module 201 is used to fill the samples of the first time period sample set between adjacent preset sample nodes using an interpolation function to obtain corresponding variable filling values ​​and generate a corresponding number of virtual samples;

[0121] In one embodiment, the interpolation filling module 201 is specifically used to use a cubic spline interpolation function to sequentially fill in the sample model points of the first time period sample set of adjacent sample nodes; obtain a corresponding number of variable filling values ​​after filling the first time period sample set; and generate a corresponding number of virtual samples according to the variable filling values.

[0122] A new sample generation module 202 is used to fill the virtual sample into the second period sample set to obtain a new sample set for the second period;

[0123] The sample variable module 203 is used to sort the sample variables of the sample set of the first period and the sample variables of the new sample set of the second period according to a preset variable contribution sorting method, and obtain the first sample variable set and the second sample new variable set in corresponding order;

[0124] The sample variable module 203 is specifically used for:

[0125] Sorting the sample variables of the first time period sample set according to the preset variable contribution ranking method to obtain the first sample variable set in a corresponding order;

[0126] Sorting the sample variables of the new sample set in the second time period according to the preset variable contribution ranking method to obtain the second sample variable set in the corresponding order;

[0127] The samples of the sample set in the first time period and the samples of the new sample set in the second time period are sorted according to a preset model sorting method.

[0128] The variable replacement module 204 is used to replace the variables of the second sample new variable set with the variables of the first sample variable set one by one in the corresponding order, calculate the individual variable error, and determine the degree of model score deviation according to the variable error.

[0129] In one embodiment, the variable replacement module 204 includes:

[0130] A sample correspondence unit, used for making one-to-one correspondence between the samples of the sample set of the first time period sorted according to the preset model sorting method and the samples of the new sample set of the second time period;

[0131] The variable error unit is used to replace the variables of the second sample new variable set one by one with the variables of the first sample variable set in the corresponding order, calculate the individual variable errors, and determine the degree of model score deviation according to the variable errors.

[0132] Among them, the variable error unit is specifically used for:

[0133] Replace the variables of the second sample new variable set one by one in the corresponding order with the variables of the first sample variable set;

[0134] Calculate the individual variable error each time after the variables of the first sample variable set replace the variables of the second sample new variable set;

[0135] The errors of all the individual variables are summed to obtain the average error of the mathematical model.

[0136] In this embodiment, a model score deviation detection device is provided, which first uses an interpolation function to fill the samples of the first time period sample set between adjacent preset sample nodes, obtains the corresponding variable filling value, and generates a corresponding number of virtual samples, fills the virtual samples into the second time period sample set, obtains the second time period new sample set, sorts the sample variables of the first time period sample set and the sample variables of the second time period new sample set according to the preset variable contribution ranking method, and obtains the first sample variable set and the second sample new variable set of corresponding order, replaces the variables of the first sample variable set with the variables of the second sample new variable set one by one according to the corresponding order, calculates the error of the individual variable, and determines the degree of model score deviation according to the error of the variable. It realizes that when the model is applied to samples of different time periods, how to determine the variable with the larger error of the individual variable when the model score deviates greatly, and correspondingly analyzes the cause of the model score deviation. However, the current mainstream machine learning algorithms themselves have a weak explanation of the cause of the deviation, and each algorithm has different principles. For example, the tree model and logistic regression have different definitions of feature importance, and it is necessary to perform deviation attribution processing for different models, which leads to the technical problem of low efficiency in mathematical model calculation.

[0137] At the same time, the present application adopts a cubic spline interpolation function to sequentially fill in the sample model points of the first time period sample set of adjacent sample nodes, so that the distribution of virtual samples is smoother.

[0138] Furthermore, it is possible to quickly locate the main cause of score deviation, determine whether it is internal or external, whether the variables are stable, and whether the model needs to be iterated, thus solving the problem of no universal model score deviation analysis method at one time and not being limited to model methods.

[0139] The present application also provides another model score deviation detection device, such as Figure 3 As shown, the device 10 includes: one or more processors 110 and a memory 120 .

[0140] The processor 110 is used to complete various control logics of the device 10, and can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISCMachine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any conventional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP, and / or any other such configuration.

[0141] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the fixed asset data online processing method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, the fixed asset data online processing method in the above method embodiment is realized. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0143] Using an interpolation function to fill in samples of the first time period sample set between adjacent preset sample nodes, obtaining corresponding variable filling values, and generating a corresponding number of virtual samples;

[0144] Filling the virtual sample into the sample set of the second period to obtain a new sample set of the second period;

[0145] Sort the sample variables of the sample set of the first period and the sample variables of the new sample set of the second period according to the preset variable contribution sorting method, and obtain the first sample variable set and the second sample new variable set in the corresponding order;

[0146] The variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in the corresponding order, the individual variable errors are calculated, and the degree of model score deviation is determined according to the variable errors.

[0147] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the above steps S101 to S104 are implemented.

[0148] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0149] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc. Mathematical model offset processing In the embodiment of the present application, a mathematical model offset processing method, device, equipment and storage medium are provided. First, an interpolation function is used to fill the samples of the first time period sample set between adjacent preset sample nodes to obtain the corresponding variable filling value, and a corresponding number of virtual samples are generated, and the virtual samples are filled into the second time period sample set to obtain a new sample set for the second time period. The sample variables of the first time period sample set and the sample variables of the second time period new sample set are sorted according to the preset variable contribution ranking method, and the first sample variable set and the second sample new variable set of the corresponding order are obtained. The variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in the corresponding order, and the individual variable error is calculated. The degree of model score offset is determined according to the variable error. Since the current model is applied to samples in different time periods, if the model score deviates significantly, it is necessary to determine the variable with the larger error of the individual variable in order to analyze the cause of the deviation in the mathematical model score. However, the current mainstream machine learning algorithms themselves have a weak explanation of the cause of the deviation, and each algorithm has a different principle. For example, the tree model and logistic regression have different definitions of feature importance, and it is necessary to perform deviation attribution processing for different models, which leads to technical problems in the calculation efficiency of the mathematical model.

[0150] At the same time, the present application adopts a cubic spline interpolation function to sequentially fill in the sample model points of the first time period sample set of adjacent sample nodes, so that the distribution of virtual samples is smoother.

[0151] Furthermore, it is possible to quickly locate the main cause of score deviation, determine whether it is internal or external, whether the variables are stable, and whether the model needs to be iterated, thus solving the problem of no universal model score deviation analysis method at one time and not being limited to model methods.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0153] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0158] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A model score deviation detection method, characterized in that: include: Filling the samples of the first time period sample set between adjacent preset sample nodes with an interpolation function to obtain corresponding variable filling values, and generating a corresponding number of virtual samples according to the variable filling values; Filling the virtual sample into the sample set of the second period to obtain a new sample set of the second period; Sort the sample variables of the sample set of the first period and the sample variables of the new sample set of the second period according to the preset variable contribution sorting method, and obtain the first sample variable set and the second sample new variable set in the corresponding order; The variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in the corresponding order, the individual variable errors are calculated, and the degree of model score deviation is determined according to the variable errors.

2. The model score deviation detection method according to claim 1, characterized in that: The interpolation function is used to fill the samples of the first time period sample set between adjacent preset sample nodes to obtain corresponding variable filling values, and a corresponding number of virtual samples are generated according to the variable filling values, and the above also includes: Sorting the samples of the sample set in the first time period according to the preset model sorting method, and calculating the corresponding quantiles; Sample nodes are calculated according to the quantiles, and correspond one to one with all samples of the sample set in the first time period.

3. The model score deviation detection method according to claim 2, characterized in that: The interpolation function is used to fill the samples of the first time period sample set between adjacent preset sample nodes to obtain corresponding variable filling values, and a corresponding number of virtual samples are generated according to the variable filling values, specifically including: Using a cubic spline interpolation function to sequentially fill in the sample model points of the first time period sample set adjacent to the sample nodes; Obtain variable filling values ​​corresponding to the number of variables after filling the sample set of the first period; Generate a corresponding number of the virtual samples according to the variable filling value.

4. The model score deviation detection method according to claim 1, characterized in that: Sorting the variables of the first period sample set and the variables of the second period new sample set according to a preset variable contribution ranking method, and obtaining the first sample variable set and the second sample new variable set in corresponding order, specifically includes: Sorting the sample variables of the first time period sample set according to the preset variable contribution ranking method to obtain the first sample variable set in a corresponding order; Sorting the sample variables of the new sample set in the second time period according to the preset variable contribution ranking method to obtain the second sample variable set in the corresponding order; The samples of the sample set in the first time period and the samples of the new sample set in the second time period are sorted according to a preset model sorting method.

5. The model score deviation detection method according to claim 1, characterized in that: The variables of the first sample variable set are replaced with the variables of the second sample new variable set one by one in a corresponding order, and the individual variable errors are calculated. The degree of model score deviation is determined according to the variable errors, specifically including: Matching the samples of the sample set of the first time period sorted according to the preset model sorting method with the samples of the new sample set of the second time period one by one; The variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in the corresponding order, the individual variable errors are calculated, and the degree of model score deviation is determined according to the variable errors.

6. The model score deviation detection method according to claim 5, characterized in that: The variables of the first sample variable set are replaced one by one with the variables of the second sample new variable set in a corresponding order, and the individual variable errors are calculated, specifically including: Replace the variables of the second sample new variable set one by one in the corresponding order with the variables of the first sample variable set; Calculate the individual variable error each time after the variables of the first sample variable set replace the variables of the second sample new variable set; The errors of all the individual variables are summed to obtain the model average error.

7. The model score deviation detection method according to claim 5, characterized in that: The preset variable contribution degree sorting method is variable contribution degree ascending order or variable contribution degree descending order; The preset model score sorting method is model score ascending order or model score descending order.

8. A model score deviation detection device, characterized in that: include: An interpolation filling module, used to fill the samples of the first time period sample set between adjacent preset sample nodes using an interpolation function to obtain corresponding variable filling values, and generate a corresponding number of virtual samples according to the variable filling values; A new sample generation module, used for filling the virtual sample into the second period sample set to obtain a new sample set for the second period; A sample variable module, used to sort the sample variables of the sample set of the first period and the sample variables of the new sample set of the second period according to a preset variable contribution sorting method, and obtain the first sample variable set and the second sample new variable set in corresponding order; The variable replacement module is used to replace the variables of the second sample new variable set one by one with the variables of the first sample variable set in the corresponding order, calculate the error of the individual variable, and determine the degree of model score deviation according to the variable error.

9. A model score deviation detection device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the model score deviation detection method described in any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the model score deviation detection method according to any one of claims 1 to 7.

Images