AUTOMATED DETERMINATION OF STOPPING CONDITIONS IN ONLINE EXPERIMENTS

Abstract

In certain embodiments, a method comprises obtaining a predetermined number of data points from an online experiment by a processing device; processing the data points by the processing device to determine whether the data points possess at least one feature; selecting a first halt rule from a plurality of halt rules by the processing device in response to the data points possessing the at least one feature, wherein the first halt rule corresponds to the at least one feature; applying the first halt rule by the processing device to determine whether a first convergence criterion is satisfied; and halting the online experiment if the first convergence criterion is satisfied.

Description

BACKGROUND

[0001] An online experiment can be any experiment that collects data in real time from any source. This data can be used to derive one or more results (e.g., a prediction, a conclusion, etc.). However, collecting or otherwise obtaining data may incur one or more costs (e.g., time, money, computational resources, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Certain embodiments discussed herein are described with reference to the drawings listed below. However, the accompanying drawings only illustrate certain aspects or implementations of the embodiments described herein by way of example and are not intended to limit the scope of the claims. Fig. 1 shows a block diagram of an example system for automatically determining stopping rules for online experiments in accordance with one or more embodiments disclosed herein; Fig. 2 shows an overview of an example method for automatically determining the stopping conditions of an online experiment according to one or more embodiments disclosed herein; and Fig. 3 shows a block diagram of a computing device according to one or more embodiments of this disclosure. DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

[0003] An online experiment can be any experiment in which data is collected in real time from any source. This data can be used to derive one or more outcomes (e.g., a prediction, a conclusion, etc.). In online experiments, data can be collected dynamically for as long as necessary to understand and summarize the data to a desired level of confidence. In one or more embodiments, an online experiment is a data stream in which one or more metrics are measured or otherwise determined. For example, an online experiment may include as data the voting information during an election, and the corresponding metric is the projected winner. As another example, an online experiment may include as data various baseball statistics, and the corresponding metric is the rate at which a batter gets a hit.Another example: An online experiment might contain web server requests as data, and the corresponding metric is the 95th percentile latency. The data collected during an online experiment might be a finite sample that can be used to make statistical inferences, predictions, etc., on a larger dataset. In general, larger data samples lead to more accurate inferences, predictions, etc.

[0004] However, obtaining additional data consumes additional resources, e.g., computational resources and / or time. Therefore, it may be useful to determine a relevant stopping rule for an online experiment. In one or more embodiments, a stopping rule for an online experiment is a rule that indicates when data collection can be terminated while still allowing a conclusion or prediction with a desired level of confidence. There may be various stopping rules that can be applied to determine the best time to stop an experiment, but choosing the appropriate rule may require an understanding of the data collected and the statistical properties of the data. Often, such an understanding is not available before the data is collected, which can be an expensive and / or time-consuming process. Therefore, determining a stopping point for an online experiment can be challenging.For example, if an online experiment is terminated too early, it may require rerunning the experiment or result in insufficient data being used, leading to incorrect conclusions. Terminating the online experiment too late may result in a waste of time and resources.

[0005] The embodiments disclosed herein may automate a dynamic process for selecting stopping rules at runtime (e.g., while the data is being collected). The embodiments disclosed herein may include two primary online components: data analysis to dynamically learn the patterns in the data, and statistical models to determine an appropriate stopping rule for each pattern. In one or more embodiments, as the data is collected and analyzed, the model evaluates and rescores the data to determine its characteristics and identify an appropriate stopping rule, which is then applied to the measured data. Therefore, the embodiments disclosed herein may precode stopping rule expertise with respect to data with different characteristics and dynamically respond to changes and transient effects in the data.

[0006] Examples of stopping rules include: stopping after a certain amount of data (e.g., number of samples), which can be relatively simple but may not be representative if the data changes over time; stopping based on the width of a confidence interval of the mean of the predicted metric, which only works if the data are normally distributed or otherwise fit the distribution corresponding to the confidence interval; stopping when a null hypothesis (e.g., that a metric meets a given value) is rejected (or cannot be rejected); stopping based on Bayesian hypothesis tests (e.g., High Density Interval (HDI));Stopping based on a regression model (e.g. Gaussian mixture models, Gaussian process regression, Bayesian linear regression, etc.), where data collection is stopped when, after a certain number of samples, a mean integrated squared error with respect to an observed distribution or information-based criterion falls below a threshold or does not decrease by a threshold of newly added samples; stopping based on a clustering model, where data collection is stopped when a number of clusters describing the data no longer increases; stopping based on a classification model, where a classification model is trained on existing data to classify the dataset as either 'stop' or 'continue' to determine whether data collection should continue;Use of extreme value theory, where data collection is stopped when, after a certain number of samples, the modeled probability of observing an extreme value or outlier falls below a threshold. Other stopping rules may also be used without deviating from the scope of the embodiments described here.

[0007] In one or more embodiments, the different stopping rules may have different strengths and weaknesses, which may be related to the characteristics of the collected data and / or the predicted metric. Some rules may stop data collection too early or too late relative to the characteristics of the collected data. Thus, in one or more embodiments, the data obtained in an online experiment may be evaluated to determine the characteristics of the data and select an appropriate stopping rule based on those characteristics.

[0008] In one or more embodiments, at least a minimum amount of data is collected, which may be predetermined. The data may be evaluated to determine properties of the data, including, but not limited to: whether the data is relatively constant (e.g., the difference between maximum and minimum values ​​is less than a threshold); whether the data is generally ascending or descending (e.g., the data is monotonic); whether the data has a normal or lognormal distribution; whether the data is unimodal or multimodal; whether the data exhibits autocorrelations, etc. In one or more embodiments, a stopping rule is selected based on one or more features of the data if a relevant stopping rule exists for the features.If no stopping rule matches one or more characteristics of the data, the online experiment can continue to collect more data, with the data being re-evaluated after more data are obtained. A predetermined maximum number of data collections can be conducted without finding a suitable stopping rule; at this point, the online experiment can be terminated.

[0009] However, if a stopping rule corresponds to one or more characteristics of the data, the stopping rule can be applied to determine whether a corresponding convergence criterion is met. The convergence criterion can depend on one or more characteristics of the data. For example, if the data have a normal distribution, a confidence interval test can be applied to the estimated mean to determine convergence. Another example: If the data are autocorrelated, a block bootstrapping procedure can be applied to determine the autocorrelation period and decide whether to discontinue the procedure. Another example: If the data are multimodal, a clustering procedure can be applied to determine whether the number of clusters in the best model fit is greater than one and whether the best model fit is good enough.The embodiments disclosed herein are not limited to the above examples. In one or more embodiments, the online experiment may be stopped if the application of the stopping rule results in the determination that an appropriate convergence criterion has been met. In one or more embodiments, the online experiment may be continued if the application of the stopping rule results in the corresponding convergence criterion not being met (e.g., the data are normally distributed, but the confidence interval of the mean of the distribution has not reached a desired level), then further data may be collected, and after a predetermined amount of additional data has been collected, the data is re-evaluated to determine one or more properties and, if possible, select a relevant stopping rule.

[0010] Certain embodiments of this disclosure can enhance online experiments by dynamically determining the properties of the data in real time to automatically select a stopping rule corresponding to one or more properties of the data. The stopping rule can be applied to determine whether a convergence criterion is met, and if so, the online experiment can be terminated. Evaluating the data in real time as it is acquired can enable the selection of a stopping rule appropriate to the data, allowing the online experiment to be terminated at a time when sufficient data has been acquired to make a conclusion or prediction within a desired confidence level, but without expending unnecessary time and resources to acquire data beyond what is necessary to make a confident conclusion or prediction.

[0011] Fig. Figure 1 shows a block diagram of an example system for automatically determining stopping rules for online experiments in accordance with one or more embodiments disclosed herein. The system may include a data source 110, a network 108, and a computing device 100. Computing device 100 may include a data collector 102, a data analyzer 104, and a stopping rule applicator 106, each of which is described below.

[0012] In one or more embodiments, as used herein, a computing device (e.g., computing device 100) may be a single computing device, a set of computing devices, one or more parts of one or more computing devices, or any other physical, virtual, and / or logical grouping of computing resources. A computing device may be referred to as an apparatus. In one or more embodiments, a computing device is a device, a part of a device, or a set of devices capable of electronically processing instructions and may include, among other things: one or more processors (e.g., components with circuitry) (not shown), memory (e.g., storage (e.g., random access memory (RAM)) (not shown), input and output device(s) (not shown), non-volatile storage hardware (e.g., solid-state drives (SSDs), hard disk drives (HDDs) (not shown)), one or more physical interfaces (e.g.,network ports, storage ports) (not shown), any number of other hardware components (not shown), and / or any combination thereof.

[0013] Examples of computing devices include, but are not limited to, a server (e.g., a blade server in a blade server enclosure, a rack server in a rack, etc.), a desktop computer, a mobile device (e.g., a laptop computer, a smartphone, a personal digital assistant, a tablet computer, a vehicle-mounted computer system, and / or other mobile computing device), a storage device (e.g., a hard disk drive array, a Fibre Channel storage device, an Internet Small Computer Systems Interface (iSCSI) storage device, a tape storage device, a flash memory array, a network-attached storage device, etc.), a network device (e.g., a switch, router, multi-layer switch, etc.), a virtual machine, a virtualized computing environment, a logical container (e.g.,for one or more applications), an Internet of Things (IoT) device, an array of nodes of computing resources, a supercomputing device, a data center or a portion thereof, a digital sensor, and / or any other type of computing device having the above requirements. In one or more embodiments, some or all of the above examples may be combined to create a system of such devices, or they may be separated into separate logical devices that may collectively be referred to as a computing device. Other types of computing devices may also be used without departing from the scope of the embodiments described herein, such as the one described in . Fig. 3 and described below.

[0014] In one or more embodiments, the memory (not shown) and / or storage (not shown) of a computing device or system of computing devices may be and / or include one or more data stores for storing any number of data structures that store any amount of data (e.g., information). In one or more embodiments, a data store is any type of storage unit and / or device (e.g., a file system, a database, a spreadsheet, RAM and / or other storage mechanism or other storage medium) for storing data. Furthermore, the data storage may include multiple different storage units and / or devices. The different storage units and / or devices may be of the same type or located in the same physical location.

[0015] In one or more embodiments, any memory (not shown) and / or storage (not shown) of a computing device or system of computing devices may be considered, in whole or in part, to be a non-transitory computer-readable medium that stores software and / or firmware.

[0016] Such software and / or firmware may include instructions that, when executed by one or more processors (not shown) and / or other hardware (e.g., circuitry) of a computing device and / or system of computing devices, cause the one or more processors and / or other hardware components to perform operations in accordance with one or more embodiments described herein.

[0017] The software instructions may be in the form of computer-readable program code to perform methods, processes, etc. of the embodiments described herein, and may, for example, be stored, in whole or in part, temporarily or permanently, on a non-transitory computer-readable medium such as a compact disc (CD), a digital versatile disc (DVD), a storage device, a floppy disk, tape storage, flash memory, physical memory, or other non-transitory computer-readable medium.

[0018] Although Fig. 1 shows a computing device 100, a person having ordinary skill in the relevant art and knowing the benefits of this disclosure will recognize that the system may comprise any number of computing devices.

[0019] In one or more embodiments, the system includes data source 110. Data source 110 may be any entity or device capable of generating and / or providing data to computing device 100. As used herein, the term data refers to any information generated, collected, or otherwise obtained through the conduct of an online experiment. An online experiment may be any experiment in which data is collected in real time (e.g., live) and analyzed as it is collected, rather than after all data for the experiment has been collected. The data obtained during an online experiment may come from one or more sources, all of which may collectively be referred to as data source 110. Data source 110 may be one or more computing devices.Data source 110 may be an aggregator of data from other sources. Data source 110 may be a disaggregated group of entities, each providing data separately. The data provided by data source 110 may be obtained using any data acquisition technique. For example, data source 110 may create, collect, or otherwise obtain performance statistics (e.g., the performance of computing resources, sports performance statistics, etc.). As another example, the data source may provide data collected during an event (e.g., votes cast during an election). Other types of data may be collected by data source 110 without departing from the scope of the embodiments disclosed herein. Although . Fig. 1 shows the system with a single data source 110, the system may include any number of data sources without departing from the scope of the embodiments disclosed herein.

[0020] In one or more embodiments, the system includes a network 108. The network 108 may refer to an entire network or any portion thereof (e.g., a logical portion of the devices within a topology of devices). A network may include a data center network, a wide area network, a local area network, a wireless network, a cellular network, and / or any other suitable network that facilitates the exchange of information from one portion of the network to another. A network may be located at a single physical location or distributed across any number of physical locations. In one or more embodiments, a network may be at least partially coupled to or overlap with the Internet. In one or more embodiments, the network 108 establishes an operative connection between the data source 110 and the computing device 100. Although Fig. 1 shows an embodiment in which the network 108 operatively connects the data source 110 and the computing device 100, one of ordinary skill in the art will understand that the data source 110 and the computing device 100 may be operatively connected in any manner that enables data to be delivered to the computing device 100, which may or may not utilize a network as described above.

[0021] In one or more embodiments, computing device 100 includes data collector 102. Data collector 102 may be any hardware (e.g., circuitry), software, firmware, or any combination thereof configured to receive data (e.g., from data source 110) to be processed by other components of computing device 100. In one or more embodiments, data collector 102 receives data from data source 110 over network 108. In such embodiments, data collector 102 may be or include a network component, such as a network interface card. Data collector 102 may receive data using any other technology without departing from the scope of the embodiments disclosed herein. Data collector 102 may receive data and pass the data to other components of computing device 100.The data collector may or may not perform some degree of initial processing on the data before making it available to other components. For example, the data may be made available to other components without any initial processing (e.g., raw data). As another example, the data collector 102 may perform initial processing on the data to modify the data in some way (e.g., format the data, change the format of the data, consolidate data of different formats into a common format, transform the data into a format expected by other components of the computing device 100, etc.). Although in . Fig. 1, computing device 100 is illustrated as having a single data collector 102, computing device 100 may include any number of data collectors without departing from the scope of the embodiments disclosed herein.

[0022] In one or more embodiments, computing device 100 includes data analyzer 104. Data analyzer 104 may be any hardware (e.g., circuitry), software, firmware, or any combination thereof configured to analyze data obtained during an online experiment. In one or more embodiments, data analyzer 104 is operatively connected to data collector 102, from which data analyzer 104 may receive data to be analyzed. In one or more embodiments, data analyzer 104 is configured to analyze the data to assess whether the data exhibits one or more characteristics. Examples of such characteristics include, but are not limited to, determining whether the data is constant, whether the data is monotonic, whether the data has a normal or lognormal distribution, whether the data is modal, whether the data is autocorrelated, etc.

[0023] In one or more embodiments, data may be determined to be constant if all of the data is the same or falls within a certain threshold. For example, if the data is numeric and each data point is the same number or falls within a threshold of one percent of the same number, the data may be determined to have the property of being constant. In one or more embodiments, data may be determined to be monotonic if the data is constantly increasing or constantly decreasing (e.g., numeric data that is constantly increasing or constantly decreasing). In one or more embodiments, data may be determined to have a normal distribution if the data is Gaussian around a mean, and lognormal if the logarithm of the data points is normally distributed around a mean.Normal distributions have a mean and a standard deviation. In one or more embodiments, data may be determined to be modal if the set of data points in the data exhibits one or more modes. Data with more than one mode may be referred to as multimodal. In one or more embodiments, data may be determined to be autocorrelated if the data exhibits a recognizable pattern that repeats over time. For example, the number of users accessing a website may vary throughout a day, but data for many days may show that the number of users increases in a repeating pattern during the same part of the day.

[0024] In one or more embodiments, the data analyzer 104 is configured to analyze the data received from the data collector 102 within any time frame and / or for preconfigured data sets. For example, the data analyzer 104 may be configured to analyze the data after a certain period of time has elapsed during data collection or after a certain number of data points have been collected since the start of an online experiment. In one or more embodiments, the data analyzer 104 is configured to re-analyze the data from time to time. For example, if it is determined that the online experiment may not yet end, the data analyzer 104 may be configured to re-analyze the data after a certain number of additional data sets have been collected.The reanalysis of the data by the data analyzer 104 may be performed at preconfigured time intervals or data sets. In one or more embodiments, the data analyzer 104 is configured to reanalyze the data from time to time based at least in part on the previous analysis of the data (e.g., based on previously determined characteristics of the data). In one or more embodiments, the data analyzer 104 stops an analysis of the data after one or more characteristics of the data have been determined or when it is determined that no features can be determined from the data obtained so far (e.g., if the data is random). For example, the data analyzer 104 may test the data to determine whether the data has a first characteristic (e.g., whether the data is constant), and if the data does not have the first characteristic, test the data to determine another characteristic (e.g.,whether they are monotonic, normally distributed, modal, autocorrelated, have a long-tail distribution, etc.). In one or more embodiments, the data analyzer 104 is configured to determine whether the data has more than one feature. For example, the data analyzer 104 may be configured, upon determining that a data sample obtained during an online experiment is multimodal, to determine whether the data is also autocorrelated. In one or more embodiments, the data analyzer 104 is configured to communicate the one or more features determined about the data, or the fact that no features were determined, to a stopping rule applicator (see below).

[0025] In one or more embodiments, computing device 100 includes stopping rule applicator 106. In one or more embodiments, stopping rule applicator 106 is operatively connected to data analyzer 104. Stopping rule applicator 106 may be any hardware (e.g., circuitry), software, firmware, or any combination thereof configured to select a stopping rule based on the one or more features or lack of features of the data and apply the selected stopping rule to determine whether an appropriate convergence criterion is met.

[0026] For example, if it is determined (e.g., by the data analyzer) that the data is constant, the chosen stopping rule may be to stop after a certain amount of data has been collected, with the amount preconfigured based on the type of data being collected and how much constant data should be collected to be sure the data does not change. In such an example, the convergence criterion may be whether enough data points have been collected.

[0027] Another example: If it is determined that the data have a monotonic characteristic, the stopping rule may be to stop the online experiment when the data have increased or decreased towards a limit and are within a threshold value of the limit, or, if the data do not converge to a limit, to stop the online experiment after a certain amount of data has been obtained but does not approach a limit.

[0028] Another example: If the data are classified as normally distributed, the stopping rule may be to end the online experiment when a certain confidence interval is reached or based on the results of conducting a t-test.

[0029] Another example: If the data are found to be multimodal, a clustering procedure can be applied to the data, and the convergence criterion can be whether the number of clusters of the best model fit is greater than one and the logarithmic probability of the best model fit is below a threshold.

[0030] Another example: If the data have the property of being autocorrelated, the stopping test may consist of terminating the online experiment if the correlation occurs repeatedly over a certain number of time intervals.

[0031] Other stopping rules and associated convergence criteria may be used by the stopping rule applicator 106 without departing from the scope of the embodiments disclosed herein (e.g., stopping based on null hypothesis testing, stopping based on Bayesian hypothesis testing, stopping based on block bootstrapping when the data are autocorrelated, stopping based on a linear or Gaussian regression model, stopping based on the results of a trained machine learning model, stopping based on testing using extreme value theory, etc.).

[0032] In one or more embodiments, if no feature has been determined for the data, the stopping rule may be to terminate the online experiment after a certain amount of time has elapsed or after a certain amount of data has been collected without a feature being determined by the data analyzer 104.

[0033] Although Fig. 1 shows that the computing device 100 includes a single stopping rule applicator 106, the computing device 100 may include any number of stopping rule applicators without departing from the scope of the embodiments disclosed herein.

[0034] Although Fig. 1 shows a particular configuration of components, other configurations may be used without departing from the scope of the embodiments described herein. Although in Fig. 1 certain components are shown as part of one and the same device, each of the components may be grouped into sets of one or more components that may exist and execute as part of any number of separate and operatively connected devices. Another example is that a single component may be configured to perform all or part of the functions described in Fig. 1. Accordingly, the embodiments disclosed herein should not be limited to the configuration of the components shown in Fig. 1 shown components.

[0035] Fig. Figure 2 shows an overview of an example method for automatically determining the stopping conditions of an online experiment in accordance with one or more embodiments disclosed herein. Fig. The method shown in Figure 2 can be used, for example, by a computing device (e.g. the one shown in Fig. 1 shown computing device 100 or the one in Fig. 3) and / or any components therein (e.g., the computing device 300 shown in Fig. 1, the data collector 102, the data analyzer 104 and / or the stopping rule applicator 106).

[0036] While the various steps in the Fig. 2 are shown and described sequentially, some or all of the steps may be performed in a different order, some or all of the steps may be combined or omitted, others in Fig. 2 steps not shown may be performed additionally and / or some or all steps may be performed in parallel with other steps of Fig. 2 are executed.

[0037] In step 200, the method includes obtaining a predetermined number of data points from an online experiment. In one or more embodiments, data are obtained during the conduct of an online experiment. Discrete units of the data may be referred to as data points. In one or more embodiments, the data points are collected by a data collector (e.g., the Fig. 1) of a data processing device (e.g. the one shown in Fig. 1 shown data processing device 100 or the one in Fig. 3). The data can be obtained from a data source (e.g., from the data processing device 300 shown in Fig. 1). In one or more embodiments, the data is obtained directly from the data source. In other embodiments, the data is obtained via a network (e.g., the network illustrated in Fig. 1). In one or more embodiments, prior to analyzing the data points, the number of data points to be acquired before beginning the data analysis is determined. In one or more embodiments, the analysis begins immediately. In other embodiments, an initial threshold number of data points (e.g., ten, one hundred, one thousand, one million, etc.) is configured to be acquired before beginning the analysis. The initial threshold number of data points to be obtained before beginning the analysis may be configured as any number without departing from the scope of the embodiments disclosed herein.In one or more embodiments, the initial threshold number of data points to be obtained before beginning the analysis is based at least in part on the type of data to be obtained, the amount of time and / or resources required to conduct the online experiment, and / or the type of outcome sought (e.g., a prediction, a conclusion, an average, etc.). In one or more embodiments, the number of data points collected before analyzing the data is configurable.

[0038] In step 202, the analysis of the data begins. The analysis of the data can be carried out, for example, by a data analyzer (e.g. the one described in Fig. 1) of a data processing device (e.g. the one shown in Fig. 1 or the data processing device shown in Fig. 3). In one or more embodiments, the analysis in step 202 includes determining whether the sample of data points obtained in step 200 has a property of being constant. In one or more embodiments, the data has the property of being constant if each data point is the same (e.g., the same numerical value). In one or more embodiments, the data has the property of being constant if each data point is within a certain threshold. For example, if all data points are within two percent of a value, such as all values ​​between 98 and 102 that are within two percent of one hundred, then the data may be determined to be constant. In one or more embodiments, the method continues with step 214 if the data is determined to be constant.In one or more embodiments, if the data is not determined to be constant, proceed to step 204.

[0039] In step 204, the data analysis continues by determining whether the sample of data points is monotonic. As used herein, monotonic refers to data that has the property of continuously increasing or decreasing. In one or more embodiments, monotonic data may increase or decrease toward a limit (e.g., 4, 4.5, 4.7, 4.9, 4.95, 4.98 may increase up to a limit of 5), which may be referred to as convergence to the limit. In one or more embodiments, monotonic data may increase or decrease without a limit. In one or more embodiments, if it is determined that the data has the property of being monotonic, the method continues with step 214. In one or more embodiments, if it is determined that the data does not have a monotonic property, the method continues with step 206.

[0040] In step 206, the data analysis continues by determining whether the data is normally distributed (e.g., has a Gaussian distribution). As used herein, a normal distribution can refer either to data that is normally distributed or to data where the logarithm of the data points is normally distributed. In one or more embodiments, a normal distribution is a distribution of data about a mean for which a standard deviation can be derived. In one or more embodiments, if it is determined that the data has the property of being normally distributed, the method continues with step 214. In one or more embodiments, if it is determined that the data does not have the property of being normally distributed, the method continues with step 208.

[0041] In step 208, the data analysis continues by determining whether the data has the property of being modal. The data may be unimodal but not normally distributed. For example, the data may be unimodal with a long-tail distribution. Unimodal data may be data that has a clear peak at a most frequently occurring value (e.g., a mode). In one or more embodiments, the data may be multimodal. In one or more embodiments, multimodal data, as used herein, is data that has two or more modes. In one or more embodiments, if it is determined that the data has the property of being modal, the method continues with step 214. In one or more embodiments, if it is determined that the data does not have a modality characteristic, the method continues with step 210.

[0042] In step 210, the data analysis continues by determining whether the data has the property of being autocorrelated. In one or more embodiments, autocorrelated data is data that has a property that demonstrates some degree of similarity from one time period to another time period. For example, a time series of data points representing traffic on a website for one day may be determined to be autocorrelated if the data for one day demonstrates a similarity to the traffic pattern observed on a certain number of subsequent days (e.g., website traffic is lower each day during the nighttime in a particular country and then increases during the daytime in the particular country). In one or more embodiments, if the data is determined to have the property of being autocorrelated, the method continues to step 214.In one or more embodiments, if it is determined that the data does not exhibit a characteristic of autocorrelation, step 212 is continued.

[0043] In step 212, it is determined whether a maximum runtime for the online experiment has expired. In one or more embodiments, a maximum runtime may be a configurable threshold for how long an online experiment should continue. The threshold may be a period of time, an amount of data, or a combination thereof. The maximum runtime may be configured based on one or more different factors, including, but not limited to, the amount of resources available to conduct the online experiment (e.g., cost, computational resources, etc.), a specific amount of time (e.g., if a result is needed before a certain time, after which the result becomes irrelevant), etc. In one or more embodiments, if the maximum runtime has expired, the method continues to step 216.In one or more embodiments, the method continues with step 214 if the maximum runtime has not yet expired.

[0044] In step 214, it is determined whether a convergence criterion is met. In one or more embodiments, a convergence criterion, as used herein, is any metric that can be determined over a sample of data points that indicates that the data is capable of producing a desired result. In one or more embodiments, the convergence criterion depends at least in part on one or more properties of the data sample determined in steps 202-212.

[0045] For example, if it is determined that the data has the property of being constant, a convergence criterion may be to determine whether at least a threshold number of constant data points has been obtained. In one or more embodiments, the convergence criterion is met when the threshold number of data points has been reached.

[0046] As another example, if the data is determined to be monotonic, the convergence criterion may include a determination of whether or not the data is approaching a limit. If the monotonic data is approaching a limit, the convergence criterion may further include a determination of whether the data has approached a particular threshold of the limit. In one or more embodiments, the confidence criterion is met if the data is within the threshold. In one or more embodiments, if the data is determined not to converge toward a limit (e.g., increasing toward infinity), the convergence criterion may further include a determination of whether enough data points have been collected to be confident that the data is not increasing or decreasing toward a limit.In one or more embodiments, the convergence criterion is met when enough data points have been collected to be confident that the monotonic data does not converge to a limit.

[0047] As another example, if the data are determined to be normally distributed, the convergence criterion may include applying a hypothesis test or determining whether the width of the confidence interval for the mean or median falls below a threshold. In one or more embodiments, the convergence criterion is met if the mean or median of the normal distribution is within the desired confidence level.

[0048] For example, if the data are classified as modal, the convergence criterion may involve the application of any technique suitable for unimodal or multimodal data. Such techniques may include, among others, the use of a regression model, such as a Gaussian mixture model, Gaussian process regression, Bayesian linear regression, kernel density estimation, etc. For such models, the convergence criterion may be determining whether a mean integrated square error (MISE) with respect to the data distribution falls below a configurable threshold, or whether the MISE does not fall below the threshold after a certain number of data points have been added.Another technique that can be applied to modal data distributions is a clustering model, which determines that after a certain number of samples, the number of clusters that best describe the data no longer increases, indicating that the convergence criterion is met. Another technique that can be applied when the data is modal is the use of a machine learning model trained to determine that there is enough data to trust the result obtained with the data obtained so far. For example, a classification model can be trained on existing and / or synthetic data to predict a class of "stop collecting data" or "continue collecting data." In one or more embodiments, the convergence criterion is met when the machine learning model predicts that data collection should stop.Another technique that can be applied when the data is modal is extreme value theory, where a convergence criterion can be to determine whether, after a certain number of samples, the modeled probability of observing an extreme value (e.g., an outlier) falls below a configurable threshold. In one or more embodiments, the convergence criterion is met when the probability has fallen below the threshold.

[0049] One skilled in the art, familiar with this disclosure, will understand that other convergence criteria and / or other stopping rules or tests may be applied without exceeding the scope of the embodiments disclosed herein. In one or more embodiments, if the convergence criterion is not met, the method continues with step 218. In one or more embodiments, if the convergence criterion is met, the method continues with step 216.

[0050] In step 216, the online experiment is stopped. In one or more embodiments, the online experiment is stopped either because the data had one or more features that matched one or more stopping rules, and the application of those one or more stopping rules resulted in a convergence criterion being met, or because the maximum runtime for the online experiment has elapsed.

[0051] In step 218, the method includes obtaining additional data, after which the method returns to step 202 to reanalyze the data. The amount of additional data obtained may be configurable and may be based at least in part on the data collected so far, one or more characteristics of the data obtained so far, the stopping rule used in step 214 and the corresponding convergence criteria, the time and resources remaining for the online experiment, a combination of the aforementioned factors, etc.

[0052] Although Fig. 2 shows that a series of data tests are performed sequentially to determine whether the data exhibit certain characteristics. In one or more embodiments, data for which a characteristic has been determined may be tested to determine whether the data also exhibit one or more additional characteristics. For example, data determined to exhibit a modal property may also be tested over time to determine whether the data is also autocorrelated. Whether data is tested for a plurality of characteristics can be configured based on expertise about the type of data, the type of online experiment, the outcome sought, etc.

[0053] In one or more embodiments, obtaining additional data (e.g., as in step 218) may involve deciding what to do with the data already obtained. In one or more embodiments, the additional data is added to the data already obtained to obtain a larger dataset, which is analyzed for various characteristics as described above. For example, data that was originally found to have a constant distribution, but for which not enough data was obtained to be confident in the constant value, may be found to be monotonic after adding more data to the dataset. In one or more embodiments, at least some of the previously obtained data may be discarded. For example, the type of data, the type of online experiment, etc.dictate that a rolling window of data collected over time should be used when testing for different characteristics. For example, an event that occurs during the online experiment may change the possible characteristics of the data, so it may be appropriate to discard data from before the event when determining the characteristics of the data. Another example: It may be determined that outliers in the data should be discarded when analyzing the data.

[0054] In one or more embodiments, other techniques may be used for such a decision, although Fig. Figure 2 shows that the online experiment is stopped due to the fulfillment of a convergence criterion. For example, if it is determined that more than one feature fits the data, more than one convergence criterion can be tested. In such scenarios, a majority decision rule can be used, stopping the online experiment when the majority of the tested convergence criteria are met, or the different stopping rules and the corresponding convergence criterion can be weighted based on the observed data, or a strict policy can be enforced, requiring all relevant convergence criteria to be met to terminate the online experiment.

[0055] Fig. Figure 3 shows a block diagram of a computing device according to one or more embodiments of this disclosure. As described above, the embodiments described herein may be implemented using computing devices. For example, all or a portion of the Fig. 1 may be implemented at least in part with one or more computing devices. Computing device 300 may include one or more computer processors 302, non-persistent memory 304 (e.g., volatile memory such as RAM, cache memory, etc.), persistent storage 306 (e.g., a hard drive, an optical drive such as a CD drive or a DVD drive, flash memory, etc.), a communications interface 312 (e.g., a Bluetooth interface, an infrared interface, a network interface, an optical interface, etc.), input devices 310, output devices 308, and numerous other elements (not shown) and functions. Each of these components is described below.

[0056] In one or more embodiments, the computer processor(s) 302 may be an integrated circuit for processing instructions. The computer processor(s) may be, for example, one or more cores or micro-cores of a processor. The processor 302 may be a general-purpose processor configured to execute program code included in the software executing on the computing device 300. The processor 302 may be a special-purpose processor in which certain instructions are integrated into the processor design. Although in Fig. 3 only one processor 302 is shown, the computing device 300 may include any number of processors without departing from the scope of the embodiments disclosed herein.

[0057] Computing device 300 may also include one or more input devices 310, such as a touchscreen, a keyboard, a mouse, a microphone, a touchpad, an electronic pen, a motion sensor, or another type of input device. Input devices 310 may enable a user to interact with computing device 300. In one or more embodiments, computing device 300 may include one or more output devices 308, such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touchscreen, a cathode ray tube (CRT) monitor, a projector, or other display device), a printer, external storage, or other output device. One or more of the output devices may be the same as or different from the input device(s).The input and output device(s) may be connected locally or remotely to the computer processor(s) 302, the non-persistent memory 304, and the persistent memory 306. There are many different types of computing devices, and the above-mentioned input and output device(s) may take other forms. In some cases, multimodal systems may allow a user to provide multiple types of input / output to communicate with the computing device 300.

[0058] In addition, the communications interface 312 may facilitate connection of the computing device 300 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or another type of network) and / or to another device, such as another computing device.The communications interface 312 may perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceiver devices, including those having an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, Bluetooth® wireless signaling, BLE wireless signaling, IBEACON® wireless signaling, RFID wireless signaling, Near Field Communication (NFC) wireless signaling, Dedicated Short Range Communication (DSRC) wireless signaling, 802.11 WiFi radio signal transmission, WLAN signal transmission, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication radio signal transmission, PSTN (Public Switched Telephone Network) signal transmission, ISDN (Integrated Services Digital Network) signal transmission, 3G / 4G / 5G / LTE radio signal transmission for cellular data networks, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, radio signal transmission along the electromagnetic spectrum, or a combination thereof.The communications interface 312 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing device 300 based on the reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, among others, the U.S. GPS, the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite System (BDS), and the European Galileo GNSS. There is no limitation to operation on any particular hardware arrangement, and therefore the basic functions here can be easily replaced with improved hardware or firmware arrangements as they are developed.

[0059] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transmitting instructions and / or data. A computer-readable medium may be a non-transitory medium capable of storing data and which does not contain carrier waves and / or transitory electronic signals that propagate wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as CDs or DVDs, flash memory, storage, or memory devices.A computer-readable medium may store code and / or machine-executable instructions that may represent a procedure, function, subroutine, program, routine, subprogram, module, software package, class, or any combination of instructions, data structures, or program instructions. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, relayed, or transmitted in any manner, e.g., by sharing memory, passing messages, passing tokens, transmitting over the network, or the like.

[0060] All or a portion of the components of computing device 300 may be implemented in circuitry. For example, the components may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuitry) and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. In some aspects, the computer-readable storage devices, media, and memories may include a wired or wireless signal containing a bitstream and the like.However, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves and signals as such.

[0061] In the above description, numerous details are given as examples of embodiments described herein. Those skilled in the art (who will also benefit from this disclosure) will understand that one or more embodiments described herein may be practiced without these specific details, and that numerous variations or modifications are possible without departing from the scope of the embodiments described herein. Certain details known to those skilled in the art may be omitted in order not to obscure the description.

[0062] In the above description, specific details are set forth to provide a thorough understanding of the aspects and examples presented herein. However, one skilled in the art will appreciate that the aspects may be practiced without these specific details. For clarity, in some cases, the present technology may be represented as functional blocks, which may include devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be represented as components in block diagram form so as not to unnecessarily obscure the aspects.In other cases, known circuits, processes, algorithms, structures and techniques may be presented without unnecessary details in order not to obscure the aspects.

[0063] Individual aspects can be described above as a process or method, which are represented in the form of a flowchart, a sequence diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are complete, but may contain additional steps not included in a diagram. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. If a process corresponds to a function, its termination can correspond to the function returning to the calling function or to the main function.

[0064] Methods and techniques according to the examples described above may be implemented using computer-executable instructions stored on computer-readable media or otherwise available. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or set of functions. Portions of the computing resources used may be accessible over a network. The computer-executable instructions may be, for example, binary files, instructions in intermediate formats such as assembly language, firmware, source code, etc.Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0065] In the above description of the figures, each component described in a figure may, in various embodiments described herein, correspond to one or more equally or similarly named and / or numbered components described in another figure. For brevity, the descriptions of these components are not repeated for each figure. Therefore, each individual embodiment of the components of each figure is incorporated by reference and is understood to be optionally present in any other figure having one or more equally or similarly named and / or numbered components.Moreover, in accordance with various embodiments described herein, any description of the components of a figure is to be understood as an optional embodiment that may be implemented in addition to, in conjunction with, or in place of the embodiments described with respect to one or more like or similarly named and / or numbered components in another figure.

[0066] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not intended to imply a particular order of the elements, nor to limit an element to a single element, unless explicitly stated, e.g., by the use of the terms "before," "after," "single," and other such terms. Rather, the use of ordinal numbers serves to distinguish between the elements. For example, a first element is to be distinguished from a second element, and the first element may comprise more than one element and may follow (or precede) the second element in the order of the elements.

[0067] As used herein, the term "operatively connected" or "operative connection" means that a direct or indirect connection exists between elements / components / devices that enables the elements to interact with each other in some way. The term "operatively connected" can refer, for example, to any direct (e.g., a direct wired connection between two devices or components) or indirect (e.g., wired and / or wireless connections between any number of devices or components connecting the operatively connected devices) connection. Thus, any path over which information can be transmitted can be considered an operative connection.

[0068] The embodiments disclosed herein provide systems and methods for automating a decision process for stopping online experiments. The disclosed stopping decision process includes collecting data in real time during an online experiment, dynamically analyzing the data to determine whether the data has one or more features, selecting a stopping rule and a corresponding convergence criterion to be tested based on the one or more features of the data, and terminating the online experiment when a convergence criterion is determined to be met.In one or more embodiments, the automated stopping decision process may respond dynamically to changes in the data and allow enough data to be collected to rely on a result based on the data, and also allow data collection to be stopped, thereby avoiding unnecessary waste of time or other resources used during data collection of the online experiment.

[0069] While the embodiments discussed herein have been described with respect to a limited number of embodiments, those skilled in the art, given the benefit of this disclosure, will recognize that other embodiments may be developed that do not depart from the scope of the embodiments disclosed herein. Accordingly, the scope of the embodiments described herein should be limited only by the appended claims.

Claims

[1] A device comprising: one or more processors and one or more non-transitory computer-readable media having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to: Obtaining a predetermined number of data points from an online experiment; Processing the data points to determine whether the data points have at least one feature; in response to the data points having the at least one feature, selecting a first stopping rule from a plurality of stopping rules, the first stopping rule corresponding to the at least one feature; Applying the first stopping rule to determine whether a first convergence criterion is met; and Stop the online experiment when the first convergence criterion is met. [2] The apparatus of claim 1, wherein the execution of the instructions further causes the one or more processors to, if the first convergence criterion is not met: to obtain additional data points. [3] The apparatus of claim 2, wherein execution of the instructions further causes the one or more processors, after receiving the additional data points, to: Determine that the data points and the additional data points have no feature; and Stopping the online experiment after a maximum runtime has elapsed. [4] The apparatus of claim 1, wherein the execution of the instructions further causes the one or more processors, prior to stopping the online experiment: in response to data points having a second feature, select a second stopping rule from the plurality of stopping rules, the second stopping rule corresponding to the second feature; and apply the second stopping rule to determine whether a second convergence criterion is met, where the online experiment is stopped when the first convergence criterion and the second convergence criterion are met. [5] The apparatus of claim 1, wherein the at least one feature comprises the data points being constant, and the first convergence criterion comprises determining whether a number of the data points is above a threshold. [6] The apparatus of claim 1, wherein the at least one feature comprises the data points being monotonic, and the first convergence criterion comprises determining that the data points are within a threshold of a limit value. [7] The apparatus of claim 1, wherein the at least one feature comprises the data points having a normal distribution and the first convergence criterion comprises determining that a mean of the data points lies within a confidence interval having a certain maximum width. [8] The apparatus of claim 1, wherein the at least one feature comprises that the data points are modal, and the first convergence criterion comprises the application of a regression model or a cluster model. [9] The apparatus of claim 1, wherein the at least one feature comprises that the data points are autocorrelated. [10] Computer-implemented method comprising: Obtaining a predetermined number of data points from an online experiment by a processing device; Processing the data points by the processing device to determine whether the data points have at least one feature; Selecting, by the processing device and in response to the data points having the at least one feature, a first stopping rule from a plurality of stopping rules, the first stopping rule corresponding to the at least one feature; applying the first stopping rule by the processing device to determine whether a first convergence criterion is met; and Stop the online experiment when the first convergence criterion is met. [11] A method according to claim 10, which, if the first convergence criterion is not met, comprises: Obtaining additional data points. [12] The method of claim 11, further comprising, after obtaining the additional data points: Determining that the data points and the additional data points have no feature; and Stopping the online experiment after a maximum runtime has elapsed. [13] The method of claim 10, further comprising, before stopping the online experiment: in response to data points having a second feature, selecting a second stopping rule from the plurality of stopping rules, the second stopping rule corresponding to the second feature; and Applying the second stopping rule to determine whether a second convergence criterion is met, where the online experiment is stopped when the first convergence criterion and the second convergence criterion are met. [14] The method of claim 10, wherein the at least one feature comprises the data points being constant, and the first convergence criterion comprises determining whether a number of the data points is above a threshold. [15] The method of claim 10, wherein the at least one feature comprises the data points being monotonic, and the first convergence criterion comprises determining that the data points are within a threshold. [16] The method of claim 10, wherein the at least one feature comprises the data points having a normal distribution and the first convergence criterion comprises determining that a mean of the data points lies within a confidence interval having a certain maximum width. [17] The method of claim 10, wherein the at least one feature comprises that the data points are modal, and the first convergence criterion comprises the application of a regression model or a cluster model. [18] The method of claim 10, wherein the at least one feature comprises that the data points are autocorrelated. [19] A non-transitory computer-readable medium storing programming for execution by one or more processors, the programming comprising instructions to: Obtaining a predetermined number of data points from an online experiment by a processing device; Processing the data points by the processing device to determine whether the data points have at least one feature; Selecting, by the processing device and in response to the data points having the at least one feature, a first stopping rule from a plurality of stopping rules, wherein the first stopping rule corresponds to the at least one feature; applying the first stopping rule by the processing device to determine whether a first convergence criterion is met; and Stop the online experiment when the first convergence criterion is met. [20] The non-transitory computer-readable medium of claim 19, wherein the programming comprises further instructions to obtain additional data points if the first convergence criterion is not met.

Images