A machine learning evaluation method with timing decay adaptation and feature drift correction

Abstract

The application discloses a kind of machine learning evaluation method of timing attenuation adaptation and feature drift correction, it is related to machine learning evaluation technical field method runs in enterprise-level machine learning operation and maintenance platform, first, interface model service gateway, artificial annotation system, three kinds of data sources of business rule engine, collect the full amount of annotation historical time series data after target model goes online, according to fixed time granularity division observation time window, after data pre-processing, with the first time window data without quality problem to construct initial evaluation benchmark library.Subsequently, the joint timing attenuation coefficient of each sample is calculated, and the evaluation contribution weight of each historical sample is dynamically allocated.The application can automatically adapt the timing weight distribution strategy according to the business evolution rhythm, effectively improve the timeliness and scene matching degree of the evaluation result, accurately measure the real influence of feature drift, distinguish the pseudo performance fluctuation caused by model real performance attenuation and feature drift, and greatly reduce the computing power and manpower loss caused by nondiscriminatory model retraining.

Description

Technical Field

[0001] This invention relates to the field of machine learning evaluation technology, and in particular to a machine learning evaluation method with temporal decay adaptation and feature drift correction. Background Technology

[0002] Machine learning models have been widely deployed in various time-series dynamic business scenarios such as financial risk control, content recommendation, industrial operation and maintenance, and traffic prediction. In these scenarios, user behavior, business rules, and external environment continue to evolve over time. The full lifecycle performance evaluation of the model after its launch is a core operation and maintenance link to ensure the reliability of business decisions and reduce operational risks. As various industries continue to increase their requirements for model fault tolerance, the timeliness, accuracy, and scenario adaptability of the evaluation results have become the core standards for measuring the effectiveness of the evaluation solution.

[0003] The current mainstream basic evaluation scheme is fixed-benchmark full-sample equal-weight evaluation. This scheme uses the offline test set before the model goes live as a fixed reference benchmark, assigns the same weight to all historical labeled samples collected after the model goes live, and directly calculates common performance indicators such as accuracy and recall. The scheme is simple to implement, consumes low computing power, and is widely used in low-iteration scenarios with stable models. However, this scheme does not consider the temporal evolution characteristics of business models. Outdated samples from the past can interfere with the current performance evaluation results, and it does not introduce a feature drift correction mechanism at all. It may misjudge the prediction error caused by feature distribution shift as a model performance degradation, frequently triggering meaningless model retraining, and even causing missed or false positive events in high-risk scenarios.

[0004] Another widely used optimization scheme is fixed-rate time-series decay assessment. This scheme sets a manually preset fixed decay rate for historical samples across different time spans, with lower weights for samples further back in time. This approach adapts to the time-series changes in business to some extent, and most commercial MLOps platforms have integrated this assessment capability. However, the decay rate in this scheme is a manually fixed parameter, which cannot adapt to the dynamic evolution rhythm of different scenarios. In scenarios with sudden changes in patterns, such as major sales events, abnormal events, or cyberattacks, the fixed decay rate cannot quickly adjust the weight distribution to adapt to the current pattern. Furthermore, it does not consider the impact of feature drift, making it impossible to distinguish between the true performance degradation of the model and the pseudo-performance fluctuations caused by feature drift. The assessment bias rate is high, and operations personnel cannot conduct targeted optimizations based on the assessment results, resulting in high overall operation and maintenance costs.

[0005] In summary, existing machine learning evaluation schemes all suffer from insufficient adaptability and lack of bias correction capabilities. They cannot simultaneously cover the two core requirements of temporal evolution adaptation and feature drift correction, making it difficult to meet the model evaluation requirements in highly dynamic and high-risk scenarios. More comprehensive evaluation technologies are urgently needed to solve these problems. Summary of the Invention

[0006] This invention proposes a machine learning evaluation method with temporal decay adaptation and feature drift correction to solve the problems of poor temporal adaptability, insufficient drift correction capability, and high evaluation bias rate mentioned in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a machine learning evaluation method for temporal decay adaptation and feature drift correction, comprising the following steps: S1. Collect the labeled historical time series dataset corresponding to the target machine learning model, covering the running data of the entire life cycle after the model is launched. Divide the continuous observation time window according to the fixed time granularity adapted to the business scenario, extract the sample feature distribution, model prediction results and real labeled values ​​in each time window, and build the initial evaluation benchmark library after completing the initial removal of outliers. S2. For samples within each time window, calculate the time decay coefficient based on the time interval between the time window to which the sample belongs and the current evaluation time point. Based on the time decay coefficient, dynamically allocate the contribution weight of each historical sample in the evaluation process, construct an evaluation sample set with time weights, and avoid interference from outdated samples in the current model performance evaluation. S3. Perform dimension-by-dimensional drift detection on the features in the evaluation sample set, calculate the drift degree value of each feature dimension relative to the initial evaluation benchmark library, perform weight correction on the feature dimensions affected by drift based on the drift degree value, filter out the evaluation bias caused by feature drift, and distinguish between the true performance degradation and the pseudo performance fluctuation caused by feature distribution shift. S4. Based on the weighted evaluation sample set, calculate the comprehensive evaluation score of the target machine learning model in combination with the preset model performance evaluation indicators, and output a structured evaluation report including time series adaptability, drift correction degree, and sub-performance indicators to ensure the matching degree of the evaluation results with the current actual running scenario of the model.

[0008] Furthermore, when constructing the initial evaluation benchmark library in S1, the preprocessing process of the initially collected labeled historical time series dataset is as follows: First, the label confidence is calculated for each sample. The label confidence is obtained by weighted summation of three dimensions: the labeler's qualification weight, the consistency rate of multi-round label cross-validation, and the sample business logic verification result. Mislabeled samples with label confidence below the preset threshold are removed. At the same time, abnormal samples with feature values ​​exceeding the reasonable range of business are removed, contradictory samples with obvious conflicts between label values ​​and business rules are removed, and incomplete samples with feature value missing rate higher than the preset threshold within a single sample are removed. Subsequently, feature engineering was performed on the retained samples. Continuous features were normalized using z-score. The mean and standard deviation parameters used in the normalization process were taken entirely from the statistical values ​​of the first complete time window without anomalies after preprocessing, so as to avoid interference from subsequent incoming samples to the normalization benchmark. Discrete features were mapped to integer codes based on a pre-built business coding dictionary. Unknown discrete values ​​not included in the dictionary were uniformly mapped to specific unknown class codes to avoid process interruptions caused by unregistered values ​​in subsequent evaluation calculations. Finally, the feature distribution, sample label distribution, and initial performance values ​​of the model on each sub-index of the first complete time window without anomalies after preprocessing are stored in the initial evaluation benchmark library as fixed benchmark reference values, providing a unified reference standard for subsequent drift detection and performance comparison.

[0009] Furthermore, the calculation of the timing decay coefficient in S2 follows the exponential decay rule, specifically expressed as follows: ,in The distance from the current assessment point The temporal decay coefficient corresponding to the sample of the time interval, The decay rate parameter is preset to match the data update frequency of the target business scenario. The time interval between the time window to which the sample belongs and the current evaluation point. The value is dynamically adjusted according to the characteristics of the scenario; Larger configurations are designed for real-time trading and real-time recommendation scenarios with high data update frequency and rapid model iteration. Values ​​are set to accelerate the weight decay of long-term samples; smaller values ​​are set for monthly forecasting and annual evaluation scenarios with low data update frequency and strong pattern stability. This value preserves the reference value of more long-term samples; The time decay coefficient is calculated using a time window as the smallest unit. All samples within the same time window share the same time decay coefficient, avoiding unnecessary computational costs associated with calculating each sample individually. Samples with shorter time intervals from the current evaluation point have larger time decay coefficients and higher contribution weights. The entire weight allocation process does not change the correspondence between the sample's own features and labels, but only adjusts the contribution ratio of samples from different time windows to the final evaluation result.

[0010] Furthermore, when dynamically allocating the contribution weight of each historical sample based on the time-series decay coefficient, the process also includes linear normalization of the time-series decay coefficient of the full evaluation samples. The initial contribution weight of each sample is equal to the time-series decay coefficient corresponding to that sample divided by the sum of the time-series decay coefficients of the full evaluation samples, so that the sum of the contribution weights of all samples is constant at 1. This avoids the weight distribution shift caused by differences in time window span and fluctuations in the sample size of different time windows, and ensures the numerical stability of the evaluation calculation process. After normalization, an upper limit is set for the contribution weight of a single sample. The contribution weight of a single sample shall not exceed three times the average contribution weight of all samples. This is to avoid the excessive weighting of the sample due to the large sample size in the time window immediately adjacent to the current evaluation point, which would lead to the evaluation results being overly dependent on very short-term data and resulting in overfitting. For the portion of sample weights that exceed the weight limit, the weights are redistributed evenly to the samples in the preceding time window in order of time from most recent to furthest, ensuring a smooth transition in weight distribution. After all weights are distributed, the total weights of all samples and the weight range of a single sample are automatically checked to see if they conform to preset rules. Only after the check passes are the weighted evaluation sample set passed to the subsequent processing module, thus avoiding numerical calculation errors in the process.

[0011] Furthermore, in S3, when performing dimension-wise drift detection on features, the maximum mean difference algorithm is used to calculate the distribution distance of each feature dimension between the current evaluation sample set and the initial evaluation benchmark library. The Gaussian kernel function is selected as the kernel function in the calculation process. The bandwidth parameter of the kernel function is preset according to the distribution characteristics of the corresponding feature dimension in the initial evaluation benchmark library to ensure the accuracy of the distribution distance calculation. Drift detection uses a single feature dimension as the smallest detection unit. The detection process does not perform dimensionality reduction on high-dimensional features to avoid the loss of single-dimensional drift information caused by dimensionality reduction operations and to prevent local weak drift features from being masked by the overall distribution index. The calculated original distribution distance is divided by the maximum distribution distance reference value obtained in the pre-experiment stage for that feature dimension, and the distribution distance is linearly normalized to a value range of 0 to 1. The normalized value is used as the drift degree value of the corresponding feature dimension. The larger the drift degree value, the higher the degree of deviation of the feature dimension relative to the baseline distribution. The entire drift detection process will synchronously record the time series of the drift degree of each feature dimension changing with the time window, providing complete data support for subsequent drift trend prediction and drift cause tracing.

[0012] Furthermore, when correcting the weights of feature dimensions affected by drift based on the drift degree value, a corresponding drift tolerance threshold is set in advance according to the sensitivity of the business scenario to feature changes: a lower drift tolerance threshold is set for financial risk control and medical diagnosis scenarios with high risk sensitivity and small decision-making error tolerance space, so as to improve the detection sensitivity of feature drift. For content recommendation and traffic prediction scenarios with rapid pattern iteration and frequent feature changes, a high drift tolerance threshold is set to avoid distortion of evaluation results caused by overcorrection. When the drift degree value of a certain feature dimension is lower than the drift tolerance threshold, it is determined that the feature has not undergone substantial drift that affects the evaluation result, and the original evaluation weight of the feature dimension is retained. When the drift value of a certain feature dimension is higher than or equal to the drift tolerance threshold, the weight of that feature dimension in the performance index calculation is reduced proportionally according to the proportion of drift value exceeding the tolerance threshold. The entire correction process does not modify the original value of the feature itself, but only adjusts the contribution ratio of the feature to the evaluation result, and retains the original information of all features for subsequent drift cause analysis.

[0013] Furthermore, when the drift value of a certain feature dimension reaches the preset complete drift threshold, it is determined that the current distribution of the feature does not have an effective overlap with the initial baseline distribution, and the semantics or value logic of the feature itself has undergone fundamental changes. Continuing to retain the participation weight of the feature will cause serious interference to the evaluation results. At this time, the participation weight of the feature dimension is reset to zero, and the interference of the feature dimension is completely eliminated in the subsequent performance index calculation process. After processing, the feature dimension is marked as a high-risk drift feature in the final output evaluation report. The time span from the first detection of drift to reaching the complete drift threshold, the distribution change trajectory during the drift process, and the value comparison information before and after drift are recorded simultaneously to provide a reference for feature reconstruction and sample supplementation in the subsequent model iteration process. After each high-risk drift feature is marked, the proportion of high-risk drift features in all features is automatically counted. If the proportion exceeds half of the total number of features, it is determined that the current business scenario has undergone an overall change and the evaluation logic of local weight correction is no longer applicable. A prompt message for full model retraining is directly triggered.

[0014] Furthermore, the preset model performance evaluation metrics in S4 are dynamically adapted according to the task type. For classification tasks, accuracy, precision, recall, F1 score, and AUC are selected as the core calculation metrics, while for regression tasks, mean absolute error, root mean square error, and coefficient of determination are selected as the core calculation metrics. When calculating the comprehensive evaluation score, a two-layer weighted calculation is performed, combining the contribution weight of each sample and the participation weight of each feature. First, at the single feature dimension level, the evaluation samples with contribution weights are used to calculate the sub-item values ​​corresponding to each feature dimension. Then, the sub-item values ​​of all single dimensions are weighted and summarized according to the participation weight of each feature to obtain the overall performance value of the model on each evaluation indicator. Finally, the comprehensive evaluation score is obtained by combining the weights of each indicator preset by the business scenario. The weights of each indicator are dynamically adjusted according to business needs. For risk-sensitive scenarios, the weight of recall indicators is increased, and for cost-sensitive scenarios, the weight of precision indicators is increased. During the calculation process, the confidence intervals of each sub-indicator are output simultaneously. The width of the confidence interval is dynamically adjusted according to the effective sample size and the degree of drift correction. The smaller the effective sample size and the higher the proportion of drift correction, the wider the confidence interval, which is used to indicate the reliability of the evaluation results.

[0015] Furthermore, after each fixed-cycle model evaluation is completed, the verified samples and feature distribution data within the current time window that have no labeled errors are synchronously stored in the initial evaluation benchmark library. Historical benchmark data that exceeds the maximum storage time is removed according to the first-in-first-out rule, so as to realize the rolling update of the evaluation benchmark library, avoid drift detection misjudgment caused by the long-term fixed benchmark library, and ensure the adaptability of benchmark reference to business scenarios.

[0016] Furthermore, after outputting the structured evaluation report, if the comprehensive evaluation score of the target machine learning model is lower than the preset performance warning threshold, model iteration prompt information will be automatically generated, and a list of high-risk drift features and low-contribution sample intervals will be pushed to the model operation and maintenance end to support targeted iterative optimization of the model and reduce the computing power and human resources costs consumed by ineffective iterations.

[0017] Compared with existing technologies, the beneficial effects of this invention are: This invention employs a scenario-adaptive joint temporal decay mechanism that dynamically adjusts the decay rate based on the similarity between the label distribution and the initial baseline. Simultaneously, it integrates sample label confidence to adjust the contribution weight of samples in each time window. This allows for automatic adaptation of weight allocation strategies according to the pace of business evolution. In scenarios with long-term stable patterns, it retains more historical sample reference value, while rapidly increasing the weight ratio of recent effective samples in scenarios with sudden pattern changes. This effectively solves the problems of poor adaptability of fixed decay parameters and lagging evaluation results compared to business changes in existing technologies, thereby improving the timeliness and scenario matching of evaluation results.

[0018] This invention accurately measures the true impact of feature drift on evaluation results by integrating the dimension-wise drift detection of the inherent predictive contribution of features with a linear soft correction mechanism. This avoids misjudging random fluctuations of low-contribution features as substantial drifts. At the same time, it filters out evaluation biases caused by drifts through feature weight correction, clearly distinguishing between the true performance degradation of the model and pseudo-performance fluctuations caused by feature distribution shifts. This effectively solves the problems of lack of drift correction capabilities, high evaluation bias rates, and inability to support targeted operation and maintenance in existing technologies, and significantly reduces the computational and human resource losses caused by indiscriminate model retraining.

[0019] This invention adopts a modular and universal design, which supports flexible configuration of core parameters such as drift threshold, indicator weight, and benchmark update cycle according to the characteristics of business scenarios. It can adapt to the evaluation needs of various business scenarios with different risk levels and different iteration rhythms. It can be quickly integrated into the mainstream MLOps operation and maintenance system without modifying the deployment architecture of the existing machine learning model. It has high promotion value and implementation feasibility, and can effectively improve the operation and maintenance efficiency and operational reliability of machine learning models across the industry. Attached Figure Description

[0020] Figure 1This is a flowchart of the overall machine learning evaluation process for temporal decay adaptation and feature drift correction proposed in this invention. Figure 2 This is a flowchart illustrating the initial benchmark library construction and temporal decay weight allocation process of this invention. Figure 3 This is a flowchart of the dimensional feature drift detection and weight correction process of this invention; Figure 4 This is a flowchart of the high-risk drift feature marking and retraining trigger determination process of the present invention; Figure 5 This is a flowchart of the calculation and early warning feedback of the dual-layer weighted performance index of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Reference Figures 1 to 5 A machine learning evaluation method for temporal decay adaptation and feature drift correction includes the following steps: S1. Collect all labeled historical time-series data of the target machine learning model from the start of its online operation. Divide the continuous observation time window into fixed time granularity adapted to the business scenario. After completing the preliminary verification and removal of mislabeled, outlier and missing values ​​for each sample, build an initial evaluation benchmark library based on the data of the first time window after the model goes online without data quality problems, so as to provide a unified reference benchmark for subsequent performance comparison and drift detection. S2. Using a single observation time window as the smallest calculation unit, the time interval between the time window to which the sample belongs and the current evaluation time point, the confidence level of the sample itself, and the scene adaptive decay rate are integrated to calculate the joint temporal decay coefficient corresponding to each sample. Based on the decay coefficient, the contribution weight of each historical sample in the evaluation calculation is dynamically allocated to construct an evaluation sample set with temporal weights. This adapts to the temporal evolution law of the business model from the sample weight level and reduces the interference of outdated samples in the past on the current performance evaluation. S3. Perform dimensional feature drift detection on the evaluation sample set with time-series weights, integrate the distribution shift of features with the inherent contribution of features to model prediction to calculate the true drift value of each feature, set hierarchical participation weights for features according to the drift degree, and perform soft correction or hard removal on features affected by random fluctuations and semantic drift respectively, correct the evaluation bias caused by feature distribution shift from the feature weight level, and distinguish between the actual performance decline of the model and the pseudo performance fluctuation caused by feature drift. S4. Based on the sample time series contribution weight and feature drift correction weight, a two-layer weighted evaluation framework is constructed. Combined with the performance index set adapted to the task type, the comprehensive evaluation score of the model is calculated, and a structured evaluation report covering time series adaptability, drift impact, and sub-performance indexes is output to ensure the matching degree of the evaluation results with the current actual running scenario of the model.

[0023] In this invention, when constructing the initial evaluation benchmark library in S1, the preprocessing process of the collected historical time series data is as follows: First, the label confidence is calculated for each sample. The label confidence is obtained by weighting the labeler's qualifications, the consistency rate of multi-round label cross-validation, and the pass rate of sample business logic verification. Mislabeled samples with label confidence lower than the preset low confidence threshold are removed. Abnormal samples with feature values ​​exceeding the reasonable range of business values ​​are removed. Contradictory samples with label values ​​that logically conflict with preset business rules are removed. Incomplete samples with feature missing rate higher than the preset missing threshold within a single sample are removed. Subsequently, uniform feature engineering is performed on the retained valid samples. For continuous features, z-score normalization is performed using the sample mean and standard deviation of the first benchmark time window. The normalization parameters are stored in a fixed manner and are not dynamically modified as subsequent samples are added, so as to avoid the consistency of the normalization benchmark being disrupted by changes in the distribution of subsequent samples. For discrete features, an encoding dictionary is pre-built based on the values ​​within the benchmark time window and integer encoding is performed. Unknown discrete values ​​not included in the encoding dictionary are uniformly mapped to fixed unknown class codes to avoid the evaluation process being interrupted due to the failure to log in the values. Finally, the feature distribution statistics, label distribution statistics, the initial performance of the model on each performance index, and the average contribution of each feature to the model prediction within the baseline time window are stored as fixed reference values ​​in the initial evaluation benchmark library. This provides a unified reference benchmark without bias for subsequent full-process evaluation calculations, eliminating the benchmark distortion problem caused by low-quality samples from the source of data.

[0024] In this invention, the calculation of the joint temporal decay coefficient in S2 adopts a scene-adaptive exponential decay mechanism. First, the decay rate parameter for scene adaptation is dynamically calculated based on the label distribution changes of adjacent time windows. The expression for the decay rate parameter is: ;in The decay rate parameter corresponding to the current evaluation period. This is the initial evaluation of the label probability distribution vector stored in the benchmark library. This is the label probability distribution vector within the most recent complete time window immediately preceding the current evaluation point. The cosine similarity between two distribution vectors, ranging from 0 to 1, corresponds to... The value ranges from 0 to 1; subsequently, the joint temporal decay coefficient is calculated for each sample, and the expression is: ;in For the first The joint time-series decay coefficients corresponding to each sample For the first The labeled confidence level for each sample For the first The time interval between each sample's time window and the current evaluation point is measured in units of fixed time granularity. A larger value indicates that the sample is further away from the current evaluation point in time; The decay rate parameter is dynamically updated according to the tag distribution changes in each evaluation period. When the tag distribution is consistent with the initial baseline, A value of 0 means that the weights of all historical samples do not decay, thus fully preserving the reference value of samples in long-term stable scenarios. This is especially important when the label distribution differs significantly from the initial baseline. The larger the value, the faster the weight decay of long-term samples in scenarios with rapid pattern iteration. At the same time, the decay intensity is adjusted in combination with the confidence of sample annotation to avoid the excessive reduction of the weight of key historical samples with high confidence, thus solving the problem that the traditional fixed decay rate cannot adapt to the evolution rhythm of different scenarios.

[0025] In this invention, when dynamically allocating the contribution weight of each historical sample based on the joint time-series decay coefficient, the joint time-series decay coefficient of all samples is first linearly normalized. The initial contribution weight of each sample is equal to its own joint time-series decay coefficient divided by the sum of the joint time-series decay coefficients of all the samples evaluated. This ensures that the total contribution weight of all samples is constant at 1, avoids the weight distribution shift caused by the fluctuation of the sample size in different time windows, and ensures the numerical stability of the evaluation calculation process. Subsequently, an upper limit for the contribution weight of a single sample is set, which shall not exceed three times the average contribution weight of the entire sample. This is to prevent the time window adjacent to the current evaluation point from having too high a weight due to the large sample size, which would lead to the evaluation results being overly dependent on very short-term data and causing random fluctuations. For the weight overflow value generated by the sample that exceeds the weight limit, the weight is smoothly redistributed to the samples in the time window that are closer to the current evaluation time point, according to the proportion of the joint time decay coefficient of the remaining samples that have not reached the weight limit, so as to ensure that the weight distribution changes continuously with the time interval without jumps. After the weight allocation is completed, numerical verification is automatically performed. After confirming that the total weight of all samples and the weight range of a single sample meet the preset rules, the evaluation sample set with time-series contribution weight is passed to the subsequent drift detection module. This avoids numerical calculation errors and overfitting risks from the weight allocation mechanism, and balances the timeliness and stability of the evaluation results.

[0026] In this invention, when performing dimension-wise feature drift detection in S3, the maximum mean difference algorithm with sample weights is used to calculate the distribution distance of each feature dimension between the current evaluation sample set and the initial evaluation benchmark library. The kernel function is a Gaussian kernel function that is adapted to the feature distribution, and the kernel bandwidth parameter is preset based on the standard deviation of the corresponding feature in the initial evaluation benchmark library to ensure the accuracy of the distribution distance calculation. The detection process uses a single feature dimension as the smallest detection unit, without performing dimensionality reduction on high-dimensional features to avoid the loss of weak drift information caused by dimensionality reduction. The drift degree value of each feature dimension is calculated by combining the fused feature distribution distance and the feature's inherent prediction contribution, expressed as: ;in For the first The true drift value for each feature dimension ranges from 0 to 1. For the first The weighted maximum mean difference between each feature dimension and the current evaluation sample set and the initial evaluation benchmark library. The first one measured in the preliminary experimental phase Reference value for the maximum mean difference between each feature dimension and the baseline distribution under a completely random distribution. The first one stored in the initial evaluation benchmark library The normalized value of the average SHAP contribution of each feature to the model prediction ranges from 0 to 1. The drift detection process synchronously records the complete time series of the drift degree of each feature dimension as it changes with the time window, providing data support for subsequent drift trend prediction and drift cause tracing. By integrating the drift measurement method of feature contribution, it solves the problem of traditional drift detection's homogeneous measurement of all features and the tendency to misjudge random fluctuations of low-contribution features as substantial drift, thus improving the accuracy of drift detection.

[0027] In this invention, when weighting feature dimensions based on drift degree values, a hierarchical drift threshold is pre-set for each feature dimension in conjunction with the risk sensitivity of the business scenario. This includes a soft correction initiation threshold and a full drift threshold: lower threshold values ​​are set for scenarios with high risk sensitivity and small decision-making error tolerance, such as financial risk control and medical diagnosis, to improve the detection sensitivity of feature drift; higher threshold values ​​are set for scenarios with fast pattern iteration and frequent feature changes, such as content recommendation and traffic prediction, to avoid distortion of evaluation results caused by over-correction. The weights for each feature dimension are calculated using a linear soft correction rule, expressed as follows: ;in For the first The weights of each feature dimension in the subsequent evaluation calculation, with values ​​ranging from 0 to 1. For the first The soft correction threshold corresponding to each feature dimension For the first The complete drift thresholds corresponding to each feature dimension are used. The correction process does not modify the original values ​​of the features themselves, but only adjusts the contribution ratio of the features to the evaluation results. The original information of all features is retained for subsequent drift cause analysis. The linear soft correction rule avoids the jump in evaluation results caused by hard threshold switching, and ensures that the evaluation score changes continuously and smoothly with the degree of feature drift.

[0028] Furthermore, when the drift value of a certain feature dimension reaches the complete drift threshold, it is determined that the current distribution of the feature does not have effective overlap with the initial baseline distribution, the feature semantics or value logic has undergone fundamental changes, the participation weight of the feature dimension is reset to 0, and the interference of the feature is completely eliminated in subsequent performance index calculations. After processing, the feature is marked as a high-risk drift feature in the evaluation log. The time span from the first trigger of soft correction to reaching the full drift threshold, the distribution change trajectory during the drift process, and the comparison information of typical values ​​before and after drift are recorded simultaneously. This provides a reference for feature reconstruction and sample supplementation in subsequent model iterations. After each high-risk drift feature is marked, the cumulative contribution ratio of the high-risk drift feature is automatically calculated. If the cumulative contribution ratio exceeds half of the total prediction contribution of the model, it is determined that the current business scenario has undergone a holistic mode switch, and local feature weight correction can no longer guarantee the accuracy of the evaluation results. A warning prompt for full model retraining is directly triggered to avoid producing erroneous evaluation conclusions when using local correction logic when the overall scenario changes.

[0029] Furthermore, in S4, when calculating the comprehensive evaluation score, the core evaluation indicator set is dynamically adapted according to the model task type. For classification tasks, accuracy, precision, recall, F1 score, and AUC are selected as core indicators; for regression tasks, mean absolute error, root mean square error, and coefficient of determination are selected as core indicators. The comprehensive evaluation score is calculated using a sample-feature two-layer weighting mechanism, expressed as: ;in This is the overall evaluation score for the model, ranging from 0 to 1. A higher score indicates better model performance. The total number of core evaluation indicators, For the first The business weights corresponding to each evaluation indicator satisfy all The sum is 1. To assess the total sample size in the sample set, The total number of feature dimensions input to the model. The contribution weights assigned to the i-th sample after time-series decay are... For the first The participation weights are obtained by drift correction of each feature. For the first The evaluation indicators in the first Feature values ​​based on sample i in each feature dimension Authentic Labels Model predictions The normalized values ​​of the calculated sub-indicators; The calculation process synchronously outputs the confidence intervals of each sub-indicator. The width of the confidence interval is dynamically adjusted according to the effective weighted sample size and the proportion of drift correction. The smaller the effective weighted sample size and the higher the cumulative contribution of the features removed with zero weight, the wider the confidence interval. This is used to intuitively indicate the reliability of the evaluation results. The dual-layer weighting mechanism takes into account both the timeliness of the samples and the validity of the features, solving the problem that traditional single-layer weighting cannot simultaneously correct for both outdated samples and feature drift.

[0030] Furthermore, after each fixed evaluation cycle of performance calculation is completed, the samples within the current time window are quality-verified. Valid samples with acceptable confidence levels and no association with high-risk drift features are synchronously stored in the initial evaluation benchmark library. Historical benchmark data exceeding the maximum storage time of the benchmark library are removed according to the first-in-first-out rule, realizing the rolling update of the evaluation benchmark library. During the rolling update process, a benchmark contamination protection mechanism is set up. If the current evaluation cycle determines that the business scenario has undergone a holistic mode switch, the benchmark library update is suspended until the new business model is stable and the benchmark reference is reconstructed. This prevents unstable samples during the transition phase from contaminating the benchmark library and causing subsequent drift detection misjudgments. While ensuring the adaptability of the benchmark library to the current business scenario, the relative stability of the benchmark is maintained, solving the problems of decreased drift detection sensitivity due to long-term lack of updates to fixed benchmarks or benchmark failure due to unconstrained updates.

[0031] Furthermore, after the structured evaluation report is generated, if the overall evaluation score is lower than the preset performance warning threshold, the triggering cause of performance degradation will be automatically identified: if the sub-indicators corresponding to high-weight, low-drift features show a significant decline, it will be determined as a true performance degradation of the model, and prompts for model parameter fine-tuning or retraining will be generated; if the proportion of the decline in sub-indicators corresponding to high-drift features to the total decline exceeds the preset threshold, it will be determined as a pseudo-performance degradation caused by feature drift, and prompts for feature engineering updates and distribution alignment will be generated. Finally, the list of high-risk drift features, low-contribution sample intervals, and classification results of the root causes of performance degradation are pushed to the model operation and maintenance end, supporting operation and maintenance personnel to carry out targeted iterative optimization, reducing the loss of computing power and human resources costs caused by indiscriminate model retraining, and improving model operation and maintenance efficiency.

[0032] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A machine learning evaluation method for temporal decay adaptation and feature drift correction, characterized in that, Includes the following steps: S1. Collect the labeled historical time series dataset corresponding to the target machine learning model, covering the running data of the entire life cycle after the model is launched. Divide the continuous observation time window according to the fixed time granularity adapted to the business scenario, extract the sample feature distribution, model prediction results and real labeled values ​​in each time window, and build the initial evaluation benchmark library after completing the initial removal of outliers. S2. For samples within each time window, calculate the time decay coefficient based on the time interval between the time window to which the sample belongs and the current evaluation time point. Based on the time decay coefficient, dynamically allocate the contribution weight of each historical sample in the evaluation process, construct an evaluation sample set with time weights, and avoid interference from outdated samples in the current model performance evaluation. S3. Perform dimension-by-dimensional drift detection on the features in the evaluation sample set, calculate the drift degree value of each feature dimension relative to the initial evaluation benchmark library, perform weight correction on the feature dimensions affected by drift based on the drift degree value, filter out the evaluation bias caused by feature drift, and distinguish between the true performance degradation and the pseudo performance fluctuation caused by feature distribution shift. S4. Based on the weighted evaluation sample set, calculate the comprehensive evaluation score of the target machine learning model in combination with the preset model performance evaluation indicators, and output a structured evaluation report including time series adaptability, drift correction degree, and sub-performance indicators to ensure the matching degree of the evaluation results with the current actual running scenario of the model.

2. The machine learning evaluation method for temporal decay adaptation and feature drift correction according to claim 1, characterized in that, The preprocessing of the initially collected labeled historical time-series dataset during the construction of the initial evaluation benchmark library in S1 is as follows: First, the label confidence score is calculated for each sample. The label confidence score is obtained by weighted summation of three dimensions: the labeler's qualification weight, the consistency rate of cross-validation of multiple rounds of labeling, and the sample business logic verification result. Mislabeled samples with a label confidence score lower than the preset threshold are removed. At the same time, abnormal samples with feature values ​​exceeding the reasonable range of business are removed, contradictory samples with obvious conflicts between the labeled values ​​and business rules are removed, and incomplete samples with a feature value missing rate higher than the preset threshold are removed. Subsequently, feature engineering was performed on the retained samples. Continuous features were normalized using z-score. The mean and standard deviation parameters used in the normalization process were taken entirely from the statistical values ​​of the first complete time window without anomalies after preprocessing. Discrete features were mapped to integer codes based on a pre-built business coding dictionary. Unknown discrete values ​​not included in the dictionary were uniformly mapped to specific unknown class codes. Finally, the feature distribution, sample label distribution, and initial performance values ​​of the model on each sub-index of the first complete time window without anomalies after preprocessing are stored in the initial evaluation benchmark library as fixed benchmark reference values.

3. The machine learning evaluation method for temporal decay adaptation and feature drift correction according to claim 1, characterized in that, The calculation of the timing decay coefficient in S2 follows the exponential decay rule, and the specific expression is as follows: ,in The distance from the current assessment point The temporal decay coefficient corresponding to the sample of the time interval, The decay rate parameter is preset to match the data update frequency of the target business scenario. The time interval between the time window to which the sample belongs and the current evaluation point. The value is dynamically adjusted according to the characteristics of the scenario.

4. The machine learning evaluation method for temporal decay adaptation and feature drift correction according to claim 3, characterized in that, When dynamically allocating the contribution weight of each historical sample in the evaluation process based on the time-series decay coefficient, it also includes linear normalization of the time-series decay coefficient of the full evaluation samples. The initial contribution weight of each sample is equal to the time-series decay coefficient corresponding to that sample divided by the sum of the time-series decay coefficients of the full evaluation samples, and the sum of the contribution weights of all samples is constant at 1. After normalization, a single sample contribution weight upper limit is set, and the single sample contribution weight shall not exceed three times the average contribution weight of all samples. For the sample weight overflow exceeding the upper limit, the weight is redistributed evenly to the samples in the preceding time window in order from the most recent to the oldest time to ensure a smooth transition of weight distribution. After all weights are assigned, the system automatically verifies whether the total weight of all samples and the weight range of a single sample conform to the preset rules. Once the verification is successful, the weighted evaluation sample set is then passed to the subsequent processing module.

5. The machine learning evaluation method for temporal decay adaptation and feature drift correction according to claim 1, characterized in that, In S3, when performing dimension-wise drift detection on features, the maximum mean difference algorithm is used to calculate the distribution distance of each feature dimension between the current evaluation sample set and the initial evaluation benchmark library. The Gaussian kernel function is selected as the kernel function in the calculation process, and the bandwidth parameter of the kernel function is preset according to the distribution characteristics of the corresponding feature dimension in the initial evaluation benchmark library. Drift detection uses a single feature dimension as the smallest detection unit, and the detection process does not perform dimensionality reduction processing on high-dimensional features; The calculated original distribution distance is divided by the maximum distribution distance reference value obtained in the pre-experiment stage for that feature dimension, and the distribution distance is linearly normalized to the range of 0 to 1. The normalized value is used as the drift value of the corresponding feature dimension. The larger the drift value, the higher the degree of deviation of that feature dimension relative to the baseline distribution.

6. The machine learning evaluation method for temporal decay adaptation and feature drift correction according to claim 5, characterized in that, When performing weight correction on feature dimensions affected by drift based on drift degree values, a corresponding drift tolerance threshold is pre-set based on the sensitivity of the business scenario to feature changes: Set lower drift tolerance thresholds for financial risk control and medical diagnosis scenarios that are highly risk-sensitive and have limited margin for error in decision-making; Set a higher drift tolerance threshold for content recommendation and traffic prediction scenarios with rapid pattern iteration and frequent feature changes; When the drift value of a certain feature dimension is lower than the drift tolerance threshold, it is determined that the feature has not undergone substantial drift that affects the evaluation result, and the original evaluation weight of the feature dimension is retained. When the drift value of a certain feature dimension is higher than or equal to the drift tolerance threshold, the weight of that feature dimension in the performance index calculation is reduced proportionally according to the proportion of drift value exceeding the tolerance threshold. The entire correction process does not modify the original value of the feature itself, but only adjusts the contribution ratio of the feature to the evaluation result, and retains the original information of all features for subsequent drift cause analysis.

7. The machine learning evaluation method for temporal decay adaptation and feature drift correction according to claim 6, characterized in that, When the drift value of a certain feature dimension reaches the preset complete drift threshold, it is determined that the current distribution of the feature does not have effective overlap with the initial baseline distribution, and the semantics or value logic of the feature itself has undergone fundamental changes. Continuing to retain the participation weight of the feature will seriously interfere with the evaluation results. At this time, the participation weight of the feature dimension is reset to zero, and the interference of the feature dimension is completely eliminated in the subsequent performance index calculation process. After processing, the feature dimension is marked as a high-risk drift feature in the final output evaluation report. The time span from the first detection of drift to reaching the complete drift threshold, the distribution change trajectory during the drift process, and the value comparison information before and after drift are recorded simultaneously to provide a reference for feature reconstruction and sample supplementation in the subsequent model iteration process. After each high-risk drift feature is marked, the proportion of high-risk drift features in all features is automatically counted. If the proportion exceeds half of the total number of features, it is determined that the current business scenario has undergone an overall change and the evaluation logic of local weight correction is no longer applicable. A prompt message for full model retraining is directly triggered.

8. The machine learning evaluation method for temporal decay adaptation and feature drift correction according to claim 1, characterized in that, The preset model performance evaluation indicators in S4 are dynamically adapted according to the task type. For classification tasks, accuracy, precision, recall, F1 score, and AUC are selected as the core calculation indicators, while for regression tasks, mean absolute error, root mean square error, and coefficient of determination are selected as the calculation indicators. When calculating the comprehensive evaluation score, a two-layer weighted calculation is performed, combining the contribution weight of each sample and the participation weight of each feature. First, at the single feature dimension level, the sub-item index value corresponding to each feature dimension is obtained by weighted calculation using the evaluation samples with contribution weights. Then, the sub-item index values ​​of all single dimensions are weighted and summarized according to the participation weight of each feature to obtain the overall performance value of the model on each evaluation index. Finally, the comprehensive evaluation score is obtained by weighted summation of each index weight preset by the business scenario. The weight of each index is dynamically adjusted according to business needs. For risk-sensitive scenarios, the weight of recall indexes is increased, and for cost-sensitive scenarios, the weight of precision indexes is increased.

9. The machine learning evaluation method for temporal decay adaptation and feature drift correction according to claim 1, characterized in that, After each fixed-cycle model performance evaluation is completed, the verified samples and feature distribution data within the current time window that have no labeling errors are synchronously stored in the initial evaluation benchmark library. Historical benchmark data that exceeds the maximum storage time is removed according to the first-in-first-out rule, so as to realize the rolling update of the evaluation benchmark library.

10. The machine learning evaluation method for temporal decay adaptation and feature drift correction according to claim 1, characterized in that, After the structured evaluation report is output, if the comprehensive evaluation score of the target machine learning model is lower than the preset performance warning threshold, model iteration prompt information will be automatically generated, and the list of high-risk drift features and low contribution sample intervals will be pushed to the model operation and maintenance end to support the targeted iterative optimization of the model.

Images