Marker drift detection and adjustment in predictive modeling

Abstract

Techniques are provided for detecting a shift in a label and adjusting training data of a predictive model in response. In an embodiment, a first machine-learned derived model is used to generate an estimated label for each of a plurality of scoring instances. The first machine-learned derived model is trained based on a plurality of training instances using one or more machine-learning techniques, each of the plurality of training instances including an observed label. In response to detecting a shift in the observed label, for each of one or more of a plurality of segments, a portion of the training data corresponding to the segment is identified. For each training instance in a subset of a portion of the training data, the training instance is adjusted. The adjusted training instance is added to a final training data set. (Multiple) machine-learning techniques are used to train a second machine-learned derived model based on the final training data set.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning, and more particularly to automatically detecting a shift in output labels and automatically adjusting labels in training data based on the detection. Background Art

[0002] Machine learning is the study and construction of algorithms that can learn from data and make estimates about the data. Such algorithms operate by building models based on inputs in order to make data-driven estimates or decisions. Therefore, machine learning techniques are used to generate statistical models that are trained based on the history of property values ​​associated with one or more objects. The statistical model is trained based on multiple properties described herein. In machine learning terminology, these properties are called "features." In order to generate and train the statistical model, a set of features is specified and a set of training data is identified.

[0003] The accuracy of the model derived from machine learning depends largely on the quality and amount of training data. For example, if there are not enough training examples in the training data, the model will not be able to make accurate predictions for inputs that are similar (but not identical) to the training examples. As another example, if the training examples do not reflect real-world scenarios, the resulting model will not be able to make accurate predictions.

[0004] Changes in the environment for which estimates are made are natural and common. For example, a cloud service that monitors the performance and resource consumption of cloud applications may implement a model that estimates the amount of one or more types of computer resources to allocate to each cloud application based on the performance of the cloud applications. Cloud application performance may change over time in response to changes in how the cloud applications are used (e.g., which features are being utilized), changes in how often other applications and / or users rely on the cloud applications, and changes in the number of machines available for the cloud applications to execute on.

[0005] Typically, changes in the environment can cause a slight shift in the output label. This is called a shift in the label distribution. "Label" refers not only to the label of the training instance, but also to the real-world results that are unrelated to the output (estimate) of the model derived from machine learning. The input label is a label that is part of the training data, and the output label is the actual label observed in the historical results. For example, the model derived from machine learning is trained to estimate whether an entity will perform a specific action in response to the occurrence of one or more events. Moreover, about twenty entities typically perform a specific action per week, but in fact only ten entities perform the action in the last week. Therefore, there is a (output) label shift from twenty to ten. The shift in the label distribution will cause the accuracy of the model derived from machine learning to decrease. For a slight shift in the label distribution, it is sufficient to refresh the model derived from machine learning. Refreshing involves generating new training instances based on recent data and retraining the model derived from machine learning based on the new training instances and the old training instances.

[0006] However, for significant shifts in the label distribution, model refreshing may not work well because a significant change in the label distribution may indicate a large change in the feature weights or coefficients used to derive the correct label from the feature set. Therefore, the model learned from historical data is likely to provide incorrect estimates. However, in order to fully rebuild the model, there is not enough recent data to generate new training instances because most of the data was collected before the factor(s) that caused the significant label shift. Therefore, refreshing the model may still result in inaccurate estimates for newly measured scoring data.

[0007] The approaches described in this section are approaches that could be pursued, but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the attached picture:

[0009] Figure 1 is a block diagram depicting an example model training system for detecting label drift and adjusting training instances according to an embodiment;

[0010] Figure 2 is a flow chart depicting an example process for marker offset detection and adjustment in an embodiment;

[0011] Figure 3 is an example data graph depicting historical data and forecast data in an embodiment;

[0012] Figure 4 is a flowchart depicting an example process for adjusting training instances based on segment aspects in an embodiment;

[0013] Figure 5 is a block diagram illustrating a computer system upon which embodiments of the present invention may be implemented. DETAILED DESCRIPTION

[0014] In the following description, for the purpose of explanation, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the present invention can be implemented without these specific details. In other examples, well-known structures and devices are shown in block diagram form to avoid unnecessary obscurity of the present invention.

[0015] General Overview

[0016] A system and method for automatically adjusting training data in response to detecting a marked shift is provided. In one technique, historical data is automatically analyzed to generate and train a predictive model. The predictive model is used to estimate an aggregate value for a specific metric. The estimated aggregate value is compared with the actual or observed aggregate value for the specific metric. If the difference between the two aggregate values ​​is significant, a marked shift is detected and triggers adjustment of the training data, on which the machine learning derived model is trained. In related techniques, the training data is divided based on segments, and training instances of different segments are adjusted differently. For example, the importance weights of instances in one segment can be adjusted positively, while the importance weights of instances in another segment can be adjusted negatively.

[0017] Embodiments improve computer-related techniques by automatically and in a data-driven and scientific manner adjusting instances in training data to improve the accuracy of models in light of significant shifts in label distributions. Embodiments relate to a novel model processing system comprising two main components, wherein model label shift detection provides directional guidance for model label shift adjustment, and model label shift adjustment is a subsequent step of model label shift detection.

[0018] Additionally, embodiments utilize predictive models to automatically detect model marker drift, even when the primary purpose of prediction is to produce accurate estimates so that prompt action can be taken, e.g., weather prediction and economic forecasting. In contrast, the predictive model(s) described herein serve as a powerful and scientific tool for detecting target marker distribution drift. The prediction results are only an intermediate step. Using the prediction results to further identify model marker drift and adjust the model accordingly is the primary goal, which is fundamentally different from previous uses of prediction.

[0019] In addition, embodiments utilize novel segment-wise variations of model label offset adjustment. In contrast, existing model label offset adjustment methods do not consider segment factors. However, in many real-world problems, segment factors are very important because the degree of label offset may vary significantly between different segments.

[0020] System Overview

[0021] Figure 1 is a block diagram depicting an example model training system 100 for detecting label shifts and adjusting training instances in an embodiment.

[0022] The model training system 100 includes historical training data 110, a model trainer 120, a machine learning derived model 130, a historical scoring data set 140, a historical result 150, a label shift detector 160, a label shift adjuster 170, and a future scoring data set 180. The model trainer 120, the label shift detector 160, and the label shift adjuster 170 are implemented in software, hardware, or any combination of software and hardware. Each of the model trainer 120, the label shift detector 160, and the label shift adjuster 170 can be implemented on a single computing device, implemented separately on multiple computing devices, or distributed on multiple computing devices, where different functions of each component are implemented on different computing devices.

[0023] The model trainer 120 takes the historical training data 110 as input to train the machine-learned model 130. Each training instance in the historical training data 110 includes a set of feature values ​​and a label. The features and labels of the machine-learned model 130 vary depending on what is being estimated. For example, the machine-learned model 130 can estimate whether the computer resource utilization of the cloud system is about to exceed the capacity of the cloud system (in this case, the current resource utilization statistics and the current capacity statistics can be features of the model), whether a user will perform a specific (e.g., online) action in response to certain stimuli (in this case, the nature of the user and the nature of the stimulus are features of the model), or whether a weather event will occur given certain known conditions (in this case, current weather conditions (e.g., temperature, wind, humidity, air pressure) can be features of the model).

[0024] The machine learning derived model 130 can be a binary classification model (which estimates whether an entity or event belongs to one of two categories), a multi-class classification model (which estimates whether an entity or event belongs to one of multiple categories), or another type of model, such as a regression model that outputs a continuous quantity (e.g., an estimated specific dollar value for a house to be sold).

[0025] The historical scoring dataset 140 includes a plurality of scoring instances, each of which includes a set of feature values ​​that are input into the machine-learned model 130. For each scoring instance, the machine-learned model 130 calculates an estimated label or a score reflecting an estimate, whether the estimated label is a classification label or a regression label. The estimated label calculated by the machine-learned model 130 can be recorded in the appropriate scoring instance in the historical scoring dataset 140.

[0026] The model training system 100 also records actual tags or observed tags in the historical results 150. The observed tags in the historical results 150 are different from the estimated tags generated by the machine learning derived model 130 based on the input from the historical scoring data set 140. The historical results 150 indicate observed events or actual events or (e.g., user) behaviors. Each observed tag corresponds to a scoring instance in the historical scoring data set 140 (or a training instance in the historical training data 110). For example, if the estimated tag of a particular scoring instance (in the historical scoring data set 140) is a score indicating the likelihood that a particular user will perform a particular online action in response to a notification or message, and the particular user does not perform the particular online action, the observed action is recorded in the historical results 150 as a value indicating a negative result, e.g., "0". On the other hand, if the particular user does perform a particular online action, the observed action is recorded in the historical results 150 as a value indicating a positive result, e.g., "1".

[0027] Observation flags can be automatically generated by one or more processes that determine whether a certain event or action occurred. In some scenarios, observation flags are generated based on content that was not found in the dataset. For example, if there is no record of a user responding to a notification within two days of receiving the notification, an observation flag is generated and recorded indicating that the event did not occur.

[0028] The observation tags are automatically associated with the scoring instances used to generate the estimated tags. For example, if the event being estimated is a user action, the user is associated with a scoring instance identifier or with a user identifier and a model identifier. Each scoring instance is associated with a scoring instance identifier or a combination of a model identifier and a user identifier. In this way, the observation tags in the historical results 150 are mapped to (or associated with) scoring instances in the historical scoring dataset 140.

[0029] At least some of the observation tags in the historical results 150 may be used to score instances in the training instances that are not yet reflected in the historical training data 110. In other words, at least a portion of the historical results 150 may be data that is newer or "fresher" than the training instances found in the historical training data 110. For example, the historical training data 110 may include observation tags generated between January and December of a year, while the historical results 150 may include observation tags generated between October of the same year and June of the following year. Alternatively, the observation tags in the historical training data 110 may be the strict subject of the observation tags in the historical results 150.

[0030] A tag shift detector 160 (described in more detail herein) analyzes the historical results 150 to detect a shift in the distribution of observed tags, which detection is described in more detail below. The tag shift detector 160 includes a prediction model generator 162 and a prediction model 164 generated by the prediction model generator 162. Although only a single prediction model is depicted, the prediction model generator 162 can generate multiple prediction models based on the historical data, for example, one prediction model for each of a plurality of segments.

[0031] If, after analyzing the historical results 150, the label shift detector 160 detects a significant label shift, the label shift adjuster 170 (also described in more detail herein) adjusts or modifies the importance weights of the training instances in the historical training data 110 to generate adjusted training data 112. The model trainer 120 trains a new model 132 based on the adjusted training data 112. The new model 132 is applied to each scoring instance in the future scoring data set 180 (whose label is not yet known at the time of label shift detection and adjustment) to generate an output label or estimate for it.

[0032] Example Process

[0033] Figure 2 is a flow chart depicting an example process 200 for label shift detection and adjustment in an embodiment. Process 200 can be implemented by different components of model training system 100.

[0034] At block 210 , a marker shift is detected (eg, by a marker shift detector 160 ) based on the historical results 150 . The marker shift may be detected using one or more prediction models trained based on observed markers, some of which may be reflected in the historical results 150 .

[0035] At block 220, a determination is made as to whether the flag shift is significant. A flag shift may be considered "significant" if the aggregate output value is outside a certain range of values ​​or if the shift measure is above a certain threshold (e.g., if the aggregate is outside a 95% confidence interval). If the determination is negative, the process 200 proceeds to block 230, where the machine-learned derived model 130 is refreshed based on the historical scoring dataset 140 and the historical results 150. If the determination is positive, the process 200 proceeds to block 240.

[0036] At box 240, determine whether there are differences in segments. A segment is a group of one or more entities (e.g., people) that share one or more common characteristics. A segment can be defined or influenced by a set of one or more values ​​for a set of one or more features of the machine-learned model 130. For example, if the feature that defines the segment is geography, and there are five possible values ​​for geography, then there are five segments or 5 groups of people living in the corresponding geographic locations. As another example, if the set of features that define the segment includes industry and geography, and there are five possible values ​​for industry and two possible values ​​for geography, then there are 2×5=10 segments or 10 groups of people, each group sharing a unique pair of common industry-geography values.

[0037] Box 240 can be performed by the label shift detector 160 or by another component of the model training system 100. The difference in fragments refers to the fact that the shift of the label distribution between different fragments of an entity is substantially different. For example, if the overall label shift is outside the 95% confidence interval based on the historical results 150, and the size of the label shift of each fragment within the historical results 150 is similar, then there is no possibility of significant fragment differences. On the other hand, if the overall label shift is outside the 95% confidence interval based on the historical results 150, and the label shifts of half of the fragments within the historical results 150 are not outside the 95% confidence interval, then there are fragment differences.

[0038] If the determination in block 240 is negative, process 200 proceeds to block 250, where all (or most) training instances in historical training data 110 are adjusted or modified regardless of the segment. If the determination in block 240 is positive, process 200 proceeds to block 260, where the training instances in historical training data 110 are adjusted based on the segment aspects. For example, training instances corresponding to one segment are adjusted by a first amount, while training instances corresponding to another segment are adjusted by a second amount.

[0039] Marker offset detection

[0040] The marker shift detector 160 detects a shift in the distribution of observed markers over time. Detection of such a shift can be performed in one or more ways. For example, if the ratio of the values ​​of the observed markers is relatively constant over time (i.e., changes very little), a simple distinction can be made between (1) the ratio of the values ​​of the observed markers during a first time period and (2) the ratio of the values ​​of the observed markers during a second (subsequent) time period. Depending on the possible values ​​of the output marker, a shift indicator can be defined based on the difference. For example, in a binary classification scenario, a distribution of 30 / 70 represents a shift of 30 points compared to a distribution of 60 / 40. Any shift of more than 15 can be considered significant.

[0041] However, the ratio of observed marker values ​​typically varies significantly over time. Therefore, in many cases, a simple comparison between two values ​​will not be sufficient to detect significant marker shifts.

[0042] Therefore, in an embodiment, one or more prediction models are trained based on a portion of the historical training data 110 and / or a portion of the historical results 150. The prediction model is trained based on data that is time series data including a plurality of data points, each data point corresponding to a different time period, and corresponding to an aggregation of observation marks (in the historical training data 110 and / or the historical results 150) that occurred within the corresponding time period. For example, the observation marks may be aggregated daily, weekly, or monthly. The aggregation may be a sum, such as a daily sum or a weekly sum; or it may be an average / median, such as a daily average for each week or a weekly average for each month. Therefore, each data point in the time series data reflects an aggregate value.

[0043] One or more prediction models take into account historical (presumably "natural") trends reflected in changes in the distribution of observed markers. The prediction models are trained based on portions of historical training data 110 and / or historical results 150 that reflect time periods prior to a particular point in time, referred to herein as "candidate offset time points."

[0044] A candidate shift time point refers to a time point that may correspond to the beginning of a potential shift in the marker distribution. Candidate shift time points may be identified based on input from a user (e.g., a developer or data scientist of the machine-learned model 130). For example, a user may guess that a significant shift in the marker distribution has begun based on preliminary reports or data. As another example, a user reading news reports about global events may expect that the machine-learned model 130 will begin to perform poorly. Additionally or alternatively, the marker shift detector 160 automatically identifies multiple candidate shift time points. For example, every day in the past may be used as a candidate shift time point. Therefore, the marker shift detector 160 may perform shift detection daily, wherein, for each day it is performed, the marker shift detector 160 uses the week before the current day as a candidate shift time point.

[0045] Once the prediction model is trained based on observed markers generated before the candidate offset time point, the prediction model is used to generate predictions or estimates of one or more markers after the candidate offset time point. The input to the prediction model can be a number indicating the number of prediction values. For example, if the data on which the prediction model is trained is the weekly sum of the past fourteen months, the input value "three" indicates that the prediction model will generate three prediction values, each representing a weekly sum, and each prediction value is for each of the three weeks after the candidate offset time point.

[0046] The tag offset detector 160 compares the predictions to the observed values ​​based on the observed tags generated (or reflecting events or activities that occurred) after the candidate offset time point. Like the predictions, the observed values ​​can reflect aggregated data, except that the aggregated data is from the historical results 150. For example, if each prediction is a daily sum, then the observed value is also a daily sum.

[0047] If one or more of the predicted values ​​differ significantly from the corresponding observed label(s), the label shift detector 160 determines that a significant shift has occurred. The measure of significance may vary depending on the implementation. For example, if the observed value differs from the predicted value by more than 20%, then such a shift is significant. A user (e.g., an administrator of the model training system 100) may define the measure of significance.

[0048] In a related embodiment, the measure of significance depends on the accuracy of the prediction model. For example, if the error of the prediction model for historical data representing events that occurred before the candidate time point is relatively small, then even though the difference between the observed value and the predicted value may be relatively small, it may still trigger the detection of a significant event. In contrast, if the error of the prediction model time is relatively large, then the difference between the observed value and the predicted value must be relatively large in order to trigger the detection of a significant event.

[0049] Marker offset detection: On the fragment side

[0050] In an embodiment, the label shift detector 160 performs label shift detection based on the fragment aspect. A fragment corresponds to a portion of a scoring instance and / or training instance that shares one or more common feature values ​​or shares common other characteristics (related to one or more model features). For example, if the training instance corresponds to a specific user, one fragment may be all users living in North America, and another fragment may be all users living in South America. However, the only possible value for a geographic feature may be a country. Therefore, even if there is no scoring instance or training instance indicating North America as a geographic feature value, if there is a mapping between a country and North America, instances indicating countries in North America are grouped together. As another example, if the training instance corresponds to a software application, one fragment may be all applications that include two or more stateful operations, another fragment may be all applications that include only one stateful operation, and another fragment may be all applications that do not include any stateful operations. One or more features are entities or events for which they are estimated, such as users, software applications, organizations, countries, or weather phenomena. Example features for users and / or organizations include geography, industry, job function, employment status, seniority level, and job title.

[0051] In order to perform tag shift detection on a segment-by-segment basis, a prediction model is generated for each segment. The data on which each prediction model is based is limited to observation tags that correspond to the segment corresponding to the prediction model. For example, all observation tags corresponding to users in North America in the historical results 150 are analyzed to generate a time series of daily sums over a period of time. This time series of daily sums is used to train a prediction model for the North American segment. Similarly, all observation tags corresponding to users in South America in the historical results 150 are analyzed to generate a time series of daily sums over the (same) time period. This time series of daily sums is used to train a prediction model for the South American segment.

[0052] Marker Shift Detection: Exponential Smoothing Approach

[0053] In an embodiment, the label shift detector 160 (or another component of the model training system 100) implements an exponential smoothing algorithm to generate a set of prediction models. Each prediction model in the set of prediction models is a state space model and can be represented in a component form that includes three different components: error, trend, and seasonality. Each component has limited variation.

[0054] The error component has two possible variations: additive (A) and multiplicative (M). The trend component has five possible variations: none (N), additive (A), additively damped (Ad), multiplicative (M), and multiplicative damped (Md). The seasonal component has three possible variations: none (N), additive (A), and multiplicative (M). By considering the variations of all three components combined, there are a total of thirty possible forecast models.

[0055] The notation ETS(·,·,·) can be used to represent the thirty possible models. This notation helps remember the order in which the components are specified; for example, the model ETS(A, A d , M) represents a model with additive errors, additive damped trend, and multiplicative seasonality. The thirty possible models share a common component form. The common component form involves the state vector x t =(l t ,b t ,s t ,s t-1 ,……,s t-m+1 ) and the state-space equations of the form:

[0056] y t =w(x t-1 )+r(x t-1 )ε t ,

[0057] x t =f(x t-1 )+g(x t-1 )ε t ,

[0058] Among them, y1, y t ,……,y t-1 is the observed time series data; {ε t} are independent and identically distributed Gaussian variables with mean 0 and variance σ2; l t represents the level of the sequence at time t; b t represents the slope (or growth) of the series at time t; s t ,s t-1 ,……,s t-m is the seasonal component; and m is the length of the seasonal component. The state vector x t is unknown, the initial state x0=(l0,b0,s0,s -1 ,……,s -m+1 ) are considered as unknown parameters of the model, and the state vector x is estimated by the state space equation tThe formulas for w(·), r(·), f(·) and g(·) depend on the component variants. The simplest model in the exponential smoothing method is the simple exponential smoothing ETS(A,N,N). The component form of this model is

[0059] y t = l t-1 +ε t ,

[0060] l t = l t-1 +αε t ,

[0061] Among them, w(x t-1 )=f(x t-1 )=l t-1 ,r(x t-1 )=1,g(x t-1 )=α, and α is an unknown parameter.

[0062] Once the model is specified, it is relatively simple to compute the likelihood of the state-space model and obtain maximum likelihood estimates of the model parameters.

[0063] After generating all or a subset of the thirty models, models are selected by minimizing one or more selection criteria. Examples of selection criteria include AIC (Akaike Information Criterion), AICc (AIC corrected for small sample bias), and BIC (Bayesian Information Criterion). Given a collection of models, each selection criterion estimates the quality of each model relative to each of the other models.

[0064] One property of certain forecasting models (e.g., the ETS model) is the ability to compute a confidence interval for each forecast value. The confidence intervals for subsequent forecast values ​​(in time) can increase. Figure 3 is an example data graph 300 that depicts time series data (specifically, aggregate statistics over time) and confidence intervals, wherein some of the time series data relates to time points before a candidate offset time point, other data in the time series data are predicted values ​​(after the candidate offset time point) (i.e., line 305), and other data in the time series data (i.e., points below the outer shaded area 320) are based on observation markers and also relate to time points after the candidate offset time point.

[0065] The predictive model that generates the predicted values ​​is represented as ETS(M,N,M) in the data graph 300. The x-axis is time and is divided into months; while the y-axis is an aggregate statistic, which represents the number of events that occurred during each month. Although the predictive model can be generated based on monthly data, the predictive model may instead be generated during weekly or daily periods. However, averaging the events monthly eliminates significant variation in this more granular data and reduces the impact of outliers (which, if used to train the predictive model, may make the predictive model relatively inaccurate), thereby increasing any confidence intervals and therefore increasing the ability to detect significant marker shifts.

[0066] In the depicted example, the candidate offset time point is February 2020, and there are three predictions (forming line 305): one prediction for February 2020, one prediction for March 2020, and one prediction for April 2020. Data graph 300 also shows two aggregate statistics, each of which is based on observation markers involving events associated with February 2020 (i.e., in the inner shaded area 310) or March 2020 (i.e., below the outer shaded area 320).

[0067] As partially noted, data graph 300 depicts three shaded regions beginning at the candidate offset time point. Inner shaded region 310 indicates an 80% confidence interval, indicating that statistically, the prediction model is 80% confident that the observed (e.g., aggregated) value will fall within inner shaded region 310. Outer shaded region 320 indicates a 95% confidence level, indicating that statistically, the prediction model is 95% confident that the observed (e.g., aggregated) value will fall within either outer shaded region 320 or inner shaded region 310.

[0068] In an embodiment, if the aggregate statistics based on the observed marks fall outside a particular confidence interval (e.g., 95%), the mark shift detector 160 determines that there is a significant mark shift, which triggers the mark shift adjuster 170. In the example of the data curve graph 300, the second aggregate statistics after the candidate shift time point (which corresponds to March 2020) are outside the outer shaded area 320, indicating that the second aggregate statistics represent a significant mark shift or anomaly.

[0069] In a related embodiment, if multiple (e.g., consecutive) aggregate statistics based on observed marks fall outside one or more confidence levels, then the mark drift detector 160 determines that there is a significant mark drift. For example, no aggregate statistics fall outside a larger confidence interval (e.g., 95%), but two consecutive aggregate statistics fall outside a smaller (although still relatively large) confidence interval (e.g., 80%). In such a scenario, the mark drift adjuster 170 can be triggered.

[0070] Moreover, which side of the predicted value the aggregated value may fall on (e.g., greater than or less than the predicted value) can indicate whether any label offset adjustment should be made. For example, if the aggregated value is outside a specific confidence interval and greater than the corresponding predicted value, no label offset adjustment is triggered. On the other hand, if the aggregated value is outside a specific confidence interval and less than the corresponding predicted value, a label offset adjustment is triggered.

[0071] In an embodiment, the predictor 162 generates different prediction models for each of the plurality of segments. In an example of the ETS model, the prediction model for one segment may have different ETS components from the prediction model for another segment. For example, the prediction model for the first segment may be represented as ETS(M,N,M), while the prediction model for the second segment may be represented as ETS(A,N,N). In other words, thirty possible prediction models are generated for the first segment (based on the training instances corresponding to the first segment), and the prediction model represented as ETS(M,N,M) is finally selected for the first segment based on the described selection criteria. At the same time, thirty possible prediction models are generated for the second segment (based on the training instances corresponding to the second segment), and the prediction model represented as ETS(A,N,N) is finally selected for the second segment based on the same selection criteria.

[0072] Label offset adjustment

[0073] In an embodiment, the label offset adjuster 170 adjusts the training instances in the historical training data 110 in response to the label offset detector 160 detecting a large or significant label offset in at least a portion of the historical results 150. "Adjusting" or modifying the training instances can involve modifying the importance weight of the training instances or modifying the labels of the training instances. The importance weight of a training instance indicates how much the coefficients or weights of the features are adjusted during the training of the machine learning-derived model based on the training instance. The higher the importance weight, the greater the adjustment of the coefficients or weights of the features of the model. Conversely, the lower the importance weight, the less the adjustment of the coefficients or weights of the features of the model.

[0074] In an embodiment where the label is modified, only non-zero labels can be modified. For example, if the positive label is 1, the new value of the positive label is 1*w, where w can be 0 < w < 1. If the negative label is 0, the negative label remains unchanged. Alternatively, the negative label can be modified to a negative number.

[0075] There are a variety of ways to adjust or modify the training instances in the historical training data 110. For example, a ratio of (1) an aggregate statistic based on observed tags generated after a candidate offset time point and (2) a predicted value corresponding to the same time period as the aggregate statistic is calculated and applied to the importance weights in the training instances. As a specific example, if the aggregate statistic is 54 and the predicted value is 97, then a value of 54 / 97 is assigned to the importance weight of each training instance. However, such an adjustment is not statistically or mathematically reasonable.

[0076] The following mathematical formula and notation are used to formulate the label shift problem. X is a feature vector and Y is a label, where X and Y have a joint distribution p(X,Y) in the historical dataset, and q(X,Y) is a joint distribution in the scoring dataset (e.g., the future scoring dataset 180), and l is a loss function, which is defined as l:Y×Y→R + l is the loss function, whose input comes from the two-dimensional space Y×Y and whose output is in the one-dimensional space R + (i.e., non-negative real space). An example of a loss function l is l(f(X), Y) = (f(X) - Y) 2 , where l takes two values ​​f(X) and Y from the two-dimensional space Y×Y as input and produces a non-negative real number (f(X)-Y) 2 As output, where f(X) represents the estimated labeling via model f.

[0077] The goal of predictive modeling is to learn to make E X,Y～p The model f that minimizes l(f(X),Y):X→Y, where E X,Y～p l(f(X),Y) is the expectation of the loss function (f(X),Y) given X and Y that obey the joint distribution p. E means "expectation" and ~ means "obey the distribution". In the ideal case of p(X,Y)=q(X,Y), the pre-trained model f is still effective in the scoring dataset (e.g., the future scoring dataset 180). However, the pre-trained model f is invalid in scenarios where there is a label shift problem. As detected by one or more predictive models, the proportion of positive labels in the historical dataset may be much larger than that in the scoring dataset, which leads to a potential logical change in inferring labels based on features. In other words, p(Y|X)!=q(Y|X), which results in p(X,Y)!=q(X,Y). In this case, the optimal model for the scoring dataset is yes The minimization variable of , which should be different from the model f learned from the historical dataset.

[0078] right One challenge in performing the minimization is the lack of information about the distribution of Y in the scoring dataset, since in practice there are no observations of Y. However, observations from historical data can be used to help estimate the distribution of Y in the scoring dataset, and the following formula can be used to obtain the optimal model for the scoring dataset:

[0079]

[0080] Marker offset adjustment: black-box offset estimation

[0081] Systematic and mathematical techniques can be used to determine how much adjustments and modifications should be made to the importance weights in the training instances. Black-box bias estimation (BBSE) is such a technique. A key assumption in BBSE is called the label bias assumption: p(X|Y) = q(X|Y). This implies that the logic of deriving features from labels is consistent between the historical dataset and the scored dataset. Although this assumption seems reasonable in many use cases, there is a potential drawback: the label bias assumption may not hold globally across all segments, where each training instance is assigned to one of multiple segments. In fact, changes in label distributions may vary greatly between different segments. For example, in some geographic areas, people's behavior may change significantly in response to significant global changes, while people's behavior in other geographic areas may not change significantly.

[0082] Marker offset adjustment: Clip level adjustment

[0083] In an embodiment, the BBSE method is extended to consider different segments. The following mathematical expression is used to express the above-mentioned label shift assumption (ie, different segments behave differently). Assume that the feature vector X = (X c ,X s ), where X s represents one or more features corresponding to a segment (e.g., a geographic region or a geographic region and an industry) and takes values ​​from a discrete set S = {1, ..., s}, and where X c represents the remaining features in the machine-learned model 130. The label shift under the segment is assumed to be: p(X c |Y,X s )=q(X c |Y,X s ). Substituting this assumption into formula (1), we get:

[0084]

[0085] And get the optimal model The key is to estimate w s (Y): =q(Y|X s) / p(Y|X s ).

[0086] According to q(f(X)|X s ), we can deduce the following (assuming that Y takes values ​​from the discrete set K = {1, ..., k}, which means that this is a multi-class classification problem with k categories):

[0087]

[0088] In addition, we express q s (f(X)):=q(f(X)|X s ), and C p,s (f(X),Y):=[p(f(X)=i,Y=j|X s )] k×k , then w s (Y) = C p,s (f(X),Y) -1 q s (f(X)). Note that C p,s (f(X),Y) is the confusion matrix (of size k×k) of model f with distribution p in segment s, and q s (f(X)) is a k-dimensional vector and is the estimated label distribution of model f with distribution q within segment s. The value of k indicates the number of categories estimated by the machine-learned model 130. Therefore, if the machine-learned model 130 is a binary classification model, then k=2. A confusion matrix is ​​a specific table layout that allows the performance of an algorithm (typically a supervised learning algorithm) to be visualized. Each row of the matrix represents an instance in the estimated category, while each column represents an instance in the actual category (and vice versa). The name "confusion matrix" comes from the fact that the matrix makes it easy to see whether the system confuses two categories (i.e., generally mislabels one category as the other). Although confusion matrices are typically used for visualization, confusion matrices are used here to calculate w s (Y), where w s (Y) = C p，s (f(X), Y) -1 q s (f(X)).

[0089] Assume that in segment s, in the historical dataset extracted from p(X,Y) There exists (a)n s samples, and extracted from the scoring dataset q(X) There is (b)m s samples. and therefore, Symbol: "^" means "estimate". For example, Z is the unknown variable, and is an estimate of Z based on some observed samples. Finally, the model According to formula (2), the sum of the weights of the loss function is Minimize the obtained, where and are estimated as the proportion of segment s in the history dataset and in the scoring dataset, respectively.

[0090] Therefore, to calculate the amount to adjust the training instance (in the historical training data 110) associated with a segment s, the following are the inputs to the label offset adjuster 170: 1) historical training data 110; 2) validation / test data set in each segment s (s=1, ..., S): and 3) the scoring dataset in each segment s (s=1, ..., S):

[0091] Figure 4 4 is a flow chart depicting an example process 400 for adjusting training instances based on segment aspects in an embodiment. Process 400 may be implemented by label offset adjuster 170.

[0092] At block 410 , a machine-learned derived model (eg, machine-learned derived model 130 ) is trained based on training data (eg, historical training data 110 ) using one or more machine-learning techniques.

[0093] At block 420, a segment is selected from a set of segments. Initially, in a first iteration of block 420, the set of segments may include all possible segments. For example, if the segments are defined based on geographic features and there are five possible values ​​for the geographic features, there are initially five segments when process 400 begins.

[0094] At block 430, a confusion matrix of size k×k is generated in,

[0095] At block 440, a k-dimensional estimated label distribution vector is generated in, is the estimated label distribution q s Estimates.

[0096] At block 450, a k-dimensional weight vector is generated are the estimated weights for the k categories (one for each dimension) applied to the training instances in segment s.

[0097] At block 460, the proportion of selected segments s in the validation / test dataset and in the scoring dataset is estimated. and is the proportion of instances in segment s that are in the validation / test dataset (i.e., Among them, n s is the number of instances in segment s in the validation / test dataset, and is the total number of instances in the validation / test dataset). is the proportion of instances within segment s in the scoring dataset (i.e., Among them, m s is the number of instances in segment s in the scoring dataset, and is the total number of instances in the scoring dataset). It is not recommended to estimate from the training dataset. Because the weight vector is estimated based on the validation / test dataset and the estimation process should be consistent.

[0098] At block 470, the training instances in the training data corresponding to the selected segment s are adjusted For example, a portion of the historical training data 110 corresponding to the selected segment is modified by multiplying This modification may involve multiplying the importance weight of each training instance associated with the selected segment s by the above product.

[0099] Therefore, we can use the k-dimensional vector All training instances in segment s are weighted. For example, each training instance in segment s with a label Y that takes a value v (where v is one of the k values ​​in the label set {1, ..., k}) will be assigned a weight That is, the k-dimensional vector The vth element in .

[0100] Block 470 may involve including the modified training instance in the adjusted training data 112 .

[0101] At block 480, a determination is made as to whether there are any more segments that have not been selected. If there are any more segments that have not been selected, process 400 proceeds to block 420 where another segment is selected. If the determination in block 480 is negative, process 400 proceeds to block 490. When process 400 proceeds to block 490, all (or potentially all) training instances are modified.

[0102] At block 490, a new model is trained based on the adjusted or modified training data. For example, the model trainer 120 trains a new model 132 based on the adjusted training data 112. The new model may have the same feature set as the machine-learned derived model in block 410, or may have a different feature set than the machine-learned derived model. For example, some features may have been added to or removed from the feature set on which the machine-learned derived model 130 was trained. Scoring instances from the future scoring dataset 180 may then be input into the new model 132 to generate a score or estimate for each model.

[0103] Hardware Overview

[0104] According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices. The special-purpose computing device can be a hardwire for performing these technologies, or can include a digital electronic device (e.g., one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs)) that is permanently programmed to perform these technologies, or can include one or more general-purpose hardware processors that are programmed to perform these technologies according to program instructions in firmware, memory, other storage devices, or combinations. Such special-purpose computing devices can also combine custom hardwired logic units, ASICs, or FPGAs with custom programming to complete these technologies. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that incorporates hardwiring and / or program logic to implement these technologies.

[0105] For example, Figure 5 is a block diagram illustrating a computer system 500 upon which embodiments of the present invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general purpose microprocessor.

[0106] The computer system 500 also includes a main memory 506 (e.g., a random access memory (RAM) or other dynamic storage device) coupled to the bus 502 to store information and instructions to be executed by the processor 504. The main memory 506 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by the processor 504. When these instructions are stored in a non-transitory storage medium accessible to the processor 504, the computer system 500 is presented as a special-purpose machine customized to perform the operations specified in the instructions.

[0107] Computer system 500 also includes a read-only memory (508) or other static storage device coupled to bus 502 to store static information and instructions for processor 504. A storage device 510, such as a magnetic disk, optical disk, or solid-state drive, is provided and coupled to bus 502 to store information and instructions.

[0108] The computer system 500 may be coupled to a display 512 (e.g., a cathode ray tube (CRT)) via the bus 502 to display information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 to communicate information and command selections to the processor 504. Another type of user input device is a cursor control 516 (e.g., a mouse, trackball, or cursor direction keys) to communicate direction information and command selections to the processor 504 and control cursor movement on the display 512. The input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.

[0109] The computer system 500 may implement the techniques described herein using custom hardwired logic units, one or more ASICs or FPGAs, firmware, and / or program logic units that are combined with the computer system to make the computer system 500 a special purpose machine or program it as a special purpose machine. According to one embodiment, the techniques herein are performed by the computer system 500 in response to the processor 504 executing one or more sequences of one or more instructions contained in the main memory 506. Such instructions may be read into the main memory 506 from another storage medium (e.g., storage device 510). Execution of the sequences of instructions contained in the main memory 506 causes the processor 504 to perform the process steps described herein. In alternative embodiments, hardwired circuits may be used in place of or in combination with software instructions.

[0110] The term "storage media" as used herein refers to any non-transitory media that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical disks, magnetic disks, or solid-state drives, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, floppy disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips, or cassette tapes.

[0111] Storage media are distinct from but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including the wires that make up bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0112] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 may receive the data on the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on the bus 502. The bus 502 carries the data to the main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by the main memory 506 may optionally be stored on the storage device 510 before or after execution by the processor 504.

[0113] Computer system 500 also includes a communication interface 518 coupled to bus 502. Communication interface 518 provides a bidirectional data communication coupling to a network link 520 connected to a local network 522. For example, communication interface 518 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection to a corresponding type of telephone line. As another example, communication interface 518 can be a local area network (LAN) card for providing a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, communication interface 518 sends and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.

[0114] The network link 520 typically provides data communication to other data devices through one or more networks. For example, the network link 520 can provide a connection to a host 524 or to data equipment operated by an Internet Service Provider (ISP) 526 through a local network 522. The ISP 526 in turn provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 528. Both the local network 522 and the Internet 528 use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 520 and through the communication interface 518 carry the digital data to and from the computer system 500, and these signals are example forms of transmission media.

[0115] Computer system 500 can send messages and receive data (including program code) through network(s), network link 520 and communication interface 518. In the Internet example, server 530 can send the request code for the application through Internet 528, ISP 526, local network 522 and communication interface 518.

[0116] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510 or other non-volatile storage for later execution.

[0117] In the foregoing description, embodiments of the present invention have been described with reference to several specific details that may vary from implementation to implementation. Therefore, the description and drawings should be regarded as illustrative rather than restrictive. The sole and exclusive indicator of the scope of the present invention, and what the applicant intends as the scope of the present invention, is the literal and equivalent range of the set of claims issued by this application, in the form in which these claims are issued, including any subsequent corrections.

Claims

1. A method for marking offset detection and adjustment, comprising: For each of the multiple scoring instances in the scoring data set, generate an estimated label for each of the scoring instances using a first machine learning derived model, wherein the first machine learning derived model is trained based on multiple training instances using one or more machine learning techniques, and each of the multiple training instances includes an observed label; generating a prediction model based on time series data, the time series data being based on first observed labeled data corresponding to a first time period; generating a forecast using the forecast model; performing a comparison between the prediction and second observation data, the second observation data corresponding to a second time period after the first time period; Based on the comparison, detecting a shift of the observation mark; In response to detecting a shift of the observation mark, performing the following operations for each of one or more of the plurality of segments: identifying a portion of the training data corresponding to each of the segments; For each training instance in the subset of the portion of the training data, the following operations are performed: Adjusting each of the training instances to generate an adjusted training instance; adding the adjusted training instances to a final training dataset; training a second machine learning derived model based on the final training data set using the one or more machine learning techniques; The method is executed by one or more computing devices.

2. The method according to claim 1, wherein: Adjusting each of the training instances to generate the adjusted training instances includes modifying an importance weight of each of the training instances.

3. The method according to claim 1, further comprising: using the prediction model to determine a confidence interval for the prediction; Wherein, performing the comparison includes: determining whether a portion of the second observation data is outside the confidence interval; Wherein detecting a shift in the observation mark is based at least in part on determining that the portion of the second observation data is outside the confidence interval.

4. The method according to claim 1, wherein: Generating the prediction model comprises: generating a plurality of prediction models based on a plurality of time series data, wherein each prediction model in the plurality of prediction models is based on different time series data in the plurality of time series data, wherein each time series data in the plurality of time series data corresponds to a different segment in the plurality of segments; Using the prediction model to generate the prediction includes: using the plurality of prediction models to generate a plurality of predictions; Performing the comparison includes: performing a plurality of comparisons, each comparison being performed between a different one of the plurality of predictions and the time series data of the plurality of time series data; Detecting the shift of the observation mark includes detecting the shift of the observation mark of a first segment among the plurality of segments and detecting that the observation mark of a second segment among the plurality of segments is not shifted.

5. The method according to claim 4, wherein: Generating the prediction model includes: Generate multiple prediction models; The prediction model is selected among the plurality of prediction models based on the accuracy of each prediction model in the plurality of prediction models with respect to the time series data.

6. The method according to claim 1, further comprising: Identify a specific point in time; identifying first observation mark data based on the specific time point, wherein data within the first observation mark data is associated with a time before the specific time point; Second observation data is identified based on the specific time point, wherein data within the second observation data is associated with a time after the specific time point.

7. The method according to claim 6, wherein: The specific time point is specified in a user input or is automatically determined not based on a user input.

8. The method according to claim 1, wherein: Adjusting each of the training instances includes using a black-box offset estimation technique.

9. The method according to claim 1, wherein: The plurality of segments are based on one or more of: geographic region, industry, employment status, job function, seniority level, or job title.

10. One or more storage media storing instructions that, when executed by one or more processors, cause: For each of the multiple scoring instances in the scoring data set, an estimated tag is generated for each of the scoring instances using a model obtained by first machine learning, wherein: The first machine-learned model is trained using one or more machine-learning techniques based on a plurality of training instances, each of the plurality of training instances comprising an observation label; generating a prediction model based on time series data, the time series data being based on first observed labeled data corresponding to a first time period; generating a forecast using the forecast model; performing a comparison between the prediction and second observation data, the second observation data corresponding to a second time period after the first time period; Based on the comparison, detecting a shift of the observation mark; In response to detecting a shift of the observation mark, performing the following operations for each of one or more of the plurality of segments: identifying a portion of the training data corresponding to each of the segments; For each training instance in the subset of the portion of the training data, the following operations are performed: Adjusting each of the training instances to generate an adjusted training instance; adding the adjusted training instances to a final training dataset; The one or more machine learning techniques are used to train a second machine learning derived model based on the final training data set.

11. One or more storage media according to claim 10, wherein: Adjusting each of the training instances to generate the adjusted training instances includes modifying an importance weight of each of the training instances.

12. One or more storage media according to claim 10, wherein: The instructions, when executed by the one or more processors, further cause: using the prediction model to determine a confidence interval for the prediction; Wherein, performing the comparison includes: determining whether a portion of the second observation data is outside the confidence interval; Wherein detecting a shift in the observation mark is based at least in part on determining that the portion of the second observation data is outside the confidence interval.

13. One or more storage media according to claim 10, wherein: Generating the prediction model comprises: generating a plurality of prediction models based on a plurality of time series data, wherein each prediction model in the plurality of prediction models is based on different time series data in the plurality of time series data, wherein each time series data in the plurality of time series data corresponds to a different segment in the plurality of segments; Using the prediction model to generate the prediction includes: using the plurality of prediction models to generate a plurality of predictions; Performing the comparison includes: performing a plurality of comparisons, each comparison being performed between a different one of the plurality of predictions and the time series data of the plurality of time series data; Detecting the shift of the observation mark includes detecting the shift of the observation mark of a first segment among the plurality of segments and detecting that the observation mark of a second segment among the plurality of segments is not shifted.

14. The one or more storage media of claim 10, wherein: Generating the prediction model includes: Generate multiple prediction models; The prediction model is selected among the plurality of prediction models based on the accuracy of each prediction model in the plurality of prediction models with respect to the time series data.

15. The one or more storage media of claim 10, wherein: The instructions, when executed by the one or more processors, further cause: Identify a specific point in time; identifying first observation mark data based on the specific time point, wherein data within the first observation mark data is associated with a time before the specific time point; Second observation data is identified based on the specific time point, wherein data within the second observation data is associated with a time after the specific time point.

16. One or more storage media according to claim 15, wherein: The specific time point is specified in a user input or is automatically determined not based on a user input.

17. The one or more storage media of claim 10, wherein: Adjusting each of the training instances includes using a black-box offset estimation technique.

18. The one or more storage media of claim 10, wherein: The plurality of segments are based on one or more of: geographic region, industry, employment status, job function, seniority level, or job title.

Images