Automatic determination of stop conditions in online experiments

Abstract

The invention relates to automatic determination of stop conditions in online experiments. In certain embodiments, a method includes obtaining, by a processing device, a predetermined number of data points from an online experiment; processing, by the processing device, the data point to determine whether the data point exhibits at least one characteristic; selecting, by the processing device, a first stop rule of a plurality of stop rules in response to the data point exhibiting at least one characteristic, where the first stop rule corresponds to the at least one characteristic; applying, by the processing device, a first stop rule to determine whether a first convergence criterion is satisfied; and stopping the online experiment when the first convergence criterion is met.

Description

Background Art

[0001] An online experiment can be any experiment that collects data in real time from any source. Such data can be used to derive one or more results (e.g., predictions, inferences, etc.). However, collecting or otherwise obtaining data may incur one or more costs (e.g., time, money, computing resources, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Certain embodiments discussed herein will be described with reference to the drawings listed below. However, the drawings merely illustrate certain aspects or implementations of the embodiments described herein and are not meant to limit the scope of the claims.

[0003] Figure 1 A block diagram illustrating an example system for implementing automatic determination of stopping rules for online experiments according to one or more embodiments disclosed herein;

[0004] Figure 2 An overview of an example method for automatically determining a stopping condition for an online experiment according to one or more embodiments disclosed herein is shown; and

[0005] Figure 3 A block diagram of a computing device according to one or more embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0006] An online experiment can be any experiment that collects data in real time from any source. Such data can be used to derive one or more results (e.g., predictions, inferences, etc.). Online experiments can be used to dynamically collect data as long as the data needs to be understood and summarized at a desired confidence level. In one or more embodiments, an online experiment is a data stream that measures or otherwise obtains one or more metrics. As an example, an online experiment can include vote information during an election as data, and the corresponding metric is the expected winner. As another example, an online experiment can include various baseball statistics as data, and the corresponding metric is the batter's hit rate. As another example, an online experiment can include web server requests as data, and the corresponding metric is 95% of the tail latency. The data collected during an online experiment can be a limited sample, which can be used to make statistical inferences, predictions, etc. about a larger data set. Generally, larger data samples will lead to more accurate inferences, predictions, etc.

[0007] However, obtaining additional data consumes additional resources, such as computing resources and / or time. Therefore, it may be useful to determine relevant stopping rules for online experiments. In one or more embodiments, the stopping rule for an online experiment is a rule that indicates when data collection can be stopped, while still allowing inferences or predictions to be made with a desired confidence level. There may be various stopping rules that can be used to determine the best time to stop the experiment, but selecting an appropriate rule may involve an understanding of the data being collected and the statistical properties of the data. Typically, this understanding cannot be obtained before collecting the data, which can be an expensive and / or time-consuming process. Therefore, determining the stopping point of an online experiment can be challenging. For example, stopping an online experiment too early may result in having to redo the experiment, or using insufficient data, resulting in erroneous inferences. As another example, stopping an online experiment too late may result in a waste of time and resources.

[0008] Embodiments disclosed herein can dynamically automate the stopping rule selection process at runtime (e.g., when data is collected). Embodiments disclosed herein can combine two main online components: data analysis to dynamically learn patterns in the data, and statistical models to determine appropriate stopping rules for each pattern. In one or more embodiments, as data is collected and analyzed, the model evaluates and reevaluates the data to determine its characteristics in order to determine appropriate stopping rules, and applies the appropriate stopping rules to the measured data. Thus, embodiments disclosed herein can pre-encode expertise about stopping rules associated with data with different characteristics, and can dynamically react to changes and transient effects seen in the data.

[0009] Examples of stopping rules include, but are not limited to: stopping after a certain amount of data has been collected (e.g., a certain number of samples), which may be relatively simple but may not be representative if the data varies over time; stopping based on the width of the confidence interval for the mean of the metric being predicted, which may only work if the data is normally distributed or conforms to a distribution corresponding to the confidence interval; stopping when the null hypothesis (e.g., the metric satisfies a predetermined value) is rejected (or fails to be rejected); stopping based on a Bayesian hypothesis test (e.g., high density interval (HDI); stopping based on a regression model (e.g., Gaussian mixture model, Gaussian process regression, Bayesian linear regression, etc.), where the null hypothesis is rejected after a given number of samples; ... confidence interval for the mean of the metric being predicted, which may only work if the data is normally distributed or conforms to a distribution corresponding to the confidence interval; stopping based on a confidence interval for the mean of the metric being predicted, which may only work if the data is normally distributed or conforms to a distribution corresponding Thereafter, data collection is stopped when the mean integrated squared error relative to the observed distribution or information-based criterion drops below a threshold, or when there is no drop in the threshold for new samples added; stopping based on a clustering model, where data collection is stopped after the number of clusters describing the data stops increasing; stopping based on a classification model, where a classification model is trained with existing data to classify a data set as stop or continue to determine whether data collection should continue; using extreme value theory, where data collection is stopped if the modeled probability of observing an extreme value or outlier after a given number of samples is below a threshold. Other stopping rules may be used without departing from the scope of the embodiments disclosed herein.

[0010] In one or more embodiments, various stopping rules may have different strengths and weaknesses, which may be related to the characteristics of the data being collected and / or the metric being predicted. Certain rules may stop data collection too early or too late relative to the characteristics of the data being collected. Therefore, in one or more embodiments, the data obtained in the online experiment can be evaluated to determine the characteristics of the data so as to select an appropriate stopping rule based on the characteristics.

[0011] In one or more embodiments, at least a minimum amount of data is collected, which can be predetermined. The data can be evaluated to determine characteristics of the data, including but not limited to: whether the data is relatively constant (e.g., the difference between the maximum and minimum values ​​is less than a threshold); whether the data is generally rising or falling (e.g., whether the data is monotonic); whether the data has a normal distribution or a log-normal distribution; whether the data is unimodal or multimodal; whether the data exhibits autocorrelation, etc. In one or more embodiments, based on one or more characteristics of the data, if there are relevant stopping rules for these characteristics, a stopping rule is selected. If no stopping rule corresponds to one or more characteristics of the data, the online experiment can continue to collect more data and re-evaluate the data after more data is obtained. In the event that no suitable stopping rule is found, a predetermined maximum amount of data collection can be performed, at which point the online experiment can end.

[0012] However, if the stopping rule corresponds to one or more characteristics of the data, the stopping rule can be applied to determine whether the associated convergence criteria are met. The convergence criteria may depend on one or more characteristics of the data. For example, if the data presents a normal distribution, a confidence interval test can be performed on the estimated mean to detect convergence. As another example, if the data is autocorrelated, a block bootstrap technique can be applied to determine the autocorrelation period and decide to stop. As another example, if the data is multimodal, a clustering technique can be applied to determine whether the number of clusters on the best model fit is greater than 1, and whether the best model fit is good enough. The embodiments disclosed herein are not limited to the foregoing examples. In one or more embodiments, if the application of the stopping rule results in a determination that the appropriate convergence criteria have been met, the online experiment can be stopped. In one or more embodiments, if the application of the stopping rule determines that the appropriate convergence criteria are not met (e.g., the data is normally distributed, but the confidence interval of the mean of the distribution does not reach the desired level), the online experiment can continue, more data can be collected, and after a predetermined amount of additional data has been collected, the data is evaluated again to determine one or more characteristics, and if possible, the relevant stopping rule is selected.

[0013] Certain embodiments of the present disclosure may improve online experiments by dynamically determining characteristics of the data in real time so as to automatically select a stopping rule corresponding to one or more characteristics of the data. The stopping rule may be applied to determine whether a convergence criterion is met, and if so, the online experiment may be stopped. Evaluating the data in real time as it is acquired may allow the selection of a stopping rule appropriate for the data, which may allow the online experiment to be stopped when sufficient data has been acquired to make an inference or prediction within a desired confidence level, without expending unnecessary time and resources to acquire data beyond that required to make a confident inference or prediction.

[0014] Figure 1 A block diagram of an example system for implementing automatic determination of stopping rules for online experiments according to one or more embodiments disclosed herein is shown. The system may include a data source 110, a network 108, and a computing device 100. The computing device 100 may include a data collector 102, a data analyzer 104, and a stopping rule applicator 106, each of which is described below.

[0015] In one or more embodiments, as used herein, a computing device (e.g., computing device 100) may be any single computing device, a group of computing devices, one or more parts of one or more computing devices, or any other physical, virtual, and / or logical grouping of computing resources. A computing device may be referred to as an apparatus. In one or more embodiments, a computing device is any device, a part of a device, or any collection of devices capable of electronically processing instructions, and may include, but is not limited to, any of the following devices: one or more processors (e.g., components including circuits) (not shown), memory (e.g., random access memory (RAM)) (not shown), input and output devices (not shown), non-volatile storage hardware (e.g., solid state drive (SSD6), hard disk drive (HDD) (not shown), one or more physical interfaces (e.g., network ports, storage ports) (not shown), any number of other hardware components (not shown), and / or any combination thereof.

[0016] Examples of computing devices include, but are not limited to, servers (e.g., blade servers in blade server chassis, rack servers in racks, etc.), desktop computers, mobile devices (e.g., laptop computers, smart phones, personal digital assistants, tablet computers, automotive computing systems, and / or any other mobile computing devices), storage devices (e.g., disk drive arrays, Fibre Channel storage devices, Internet Small Computer System Interface (iSCSI) storage devices, tape storage devices, flash arrays, network attached storage devices, etc.), network devices (e.g., switches, routers, multilayer switches, etc.), virtual machines, virtualized computing environments, logical containers (e.g., for one or more applications), Internet of Things (IoT) devices, node arrays of computing resources, supercomputing devices, data centers or any portion thereof, digital sensors, and / or any other type of computing device having the foregoing requirements. In one or more embodiments, any or all of the above examples may be combined to create a system of such devices, or may be divided into separate logical devices, which may be collectively referred to as computing devices. Other types of computing devices may be used without departing from the scope of the embodiments described herein, such as, for example, Figure 3 The computing device shown in and described below.

[0017] In one or more embodiments, the storage (not shown) and / or memory (not shown) of a computing device or computing device system may be and / or include one or more data repositories for storing any number of data structures storing any number of data (e.g., information). In one or more embodiments, a data repository is any type of storage unit and / or device (e.g., a file system, a database, a collection of tables, RAM, and / or any other storage mechanism or medium) for storing data. In addition, a data repository may include multiple different storage units and / or devices. Multiple different storage units and / or devices may or may not be of the same type or located at the same physical location.

[0018] In one or more embodiments, any storage (not shown) and / or memory (not shown) of a computing device or computing device system may be considered, in whole or in part, as a non-transitory computer-readable medium that stores software and / or firmware.

[0019] Such software and / or firmware may include instructions that, when executed by one or more processors (not shown) and / or other hardware (e.g., circuits) of a computing device and / or computing device system, cause the one or more processors and / or other hardware components to perform operations according to one or more embodiments described herein.

[0020] The software instructions may be in the form of computer-readable program code to perform the methods, processes, etc. of the embodiments as described herein, and may, by way of example, be stored in whole or in part, temporarily or permanently, on a non-transitory computer-readable medium, such as a compact disc (CD), a digital versatile disc (DVD), a storage device, a disk, a tape storage, flash memory, a physical memory, or any other non-transitory computer-readable medium.

[0021] although Figure 1 One computing device 100 is shown, but persons of ordinary skill in the relevant art and having the benefit of this disclosure will appreciate that the system may include any number of computing devices.

[0022] In one or more embodiments, the system includes a data source 110. The data source 110 can be any entity or device capable of generating data and / or providing data to the computing device 100. As used herein, data refers to any information generated, collected, or otherwise obtained by conducting an online experiment. An online experiment can be any experiment that obtains data in real time (e.g., real time) and analyzes the data as it is obtained, rather than after all the data for the experiment is obtained. The data obtained during the online experiment can come from any one or more sources, all of which can be collectively referred to as a data source 110. The data source 110 can be one or more computing devices. The data source 110 can be an aggregator of data from other sources. The data source 110 can be a set of decomposed entities, each of which provides the obtained data separately. The data provided by the data source 110 can be obtained using any technology for obtaining data. As an example, the data source 110 can generate, collect, or otherwise obtain performance statistics (e.g., performance of computing resources, sports performance statistics, etc.). As another example, the data source can provide data obtained during an event (e.g., votes during an election). The data source 110 can obtain other types of data without departing from the scope of the embodiments disclosed herein. Although Figure 1 A system is shown including a single data source 110 , but the system may include any number of data sources without departing from the scope of the embodiments disclosed herein.

[0023] In one or more embodiments, the system includes a network 108. The network 108 can refer to the entire network or any portion thereof (e.g., a logical portion of a device within a device topology). The network can include a data center network, a wide area network, a local area network, a wireless network, a cellular telephone network, and / or any other suitable network that facilitates the exchange of information from one part of the network to another. The network can be located at a single physical location or distributed across any number of physical sites. In one or more embodiments, the network can be at least partially coupled to or overlap with the Internet. In one or more embodiments, the network 108 provides an efficient connection between the data source 110 and the computing device 100. Although Figure 1 An embodiment is shown in which a network 108 operably connects a data source 110 and a computing device 100, but one of ordinary skill in the art will understand that the data source 110 and the computing device 100 may be operably connected in any manner that allows data to be provided to the computing device 100, which may or may not use a network as described above.

[0024] In one or more embodiments, the computing device 100 includes a data collector 102. The data collector 102 can be any hardware (e.g., circuitry), software, firmware, or any combination thereof that is configured to receive data (e.g., from a data source 110) to be processed by other components of the computing device 100. In one or more embodiments, the data collector 102 receives data from the data source 110 via a network 108. In such embodiments, the data collector 102 can be or include a network component, such as a network interface card. The data collector 102 can receive data using any other technology without departing from the scope of the embodiments disclosed herein. The data collector 102 can receive data and provide the data to other components of the computing device 100. The data collector may or may not perform a certain amount of initial processing on the data before providing the data to the other components. For example, the data can be provided to the other components (e.g., raw data) without any initial processing. As another example, the data collector 102 can perform initial processing on the data in order to change the data in some way (e.g., format the data, change the format of the data, put data of various formats into a common format, convert the data into a format expected by other components of the computing device 100, etc.). Although Figure 1 The computing device 100 is shown as including a single data collector 102 , but the computing device 100 may include any number of data collectors without departing from the scope of the embodiments disclosed herein.

[0025] In one or more embodiments, the computing device 100 includes a data analyzer 104. The data analyzer 104 can be any hardware (e.g., circuitry), software, firmware, or any combination thereof configured to analyze data obtained during an online experiment. In one or more embodiments, the data analyzer 104 is operably connected to the data collector 102, and the data analyzer 104 can receive data from the data collector 102 for analysis. In one or more embodiments, the data analyzer 104 is configured to analyze the data to assess whether the data exhibits one or more characteristics. Examples of such characteristics may include, but are not limited to, determining whether the data is constant, whether the data is monotonic, whether the data has a normal or log-normal distribution, whether the data is modal, whether the data is autocorrelated, and the like.

[0026] In one or more embodiments, if all data are the same or within a certain threshold, it can be determined that the data is constant. For example, if the data is numerical and each data point is the same number, or within a threshold of one percent of the same number, it can be determined that the data exhibits constant characteristics. In one or more embodiments, if the data increases or decreases consistently (e.g., numerical data that increases or decreases consistently), it can be determined that the data is monotonic. In one or more embodiments, when the data exhibits a Gaussian distribution around the mean, the data can be determined to have a normal distribution, and when the logarithm of the data points is normally distributed around the mean, the data can be determined to be a log-normal distribution. A normal distribution has a mean and a standard deviation. In one or more embodiments, if a set of data points of the data exhibits one or more modes, it can be determined that the data is modal. Data with more than one mode can be called multimodal. In one or more embodiments, when the data exhibits a repetitive detectable mode over time, it can be determined that the data is autocorrelated. For example, the number of users visiting a website in a day may vary over time, but many days of data may show that the number of users peaks in a repetitive mode at the same time of the day.

[0027] In one or more embodiments, the data analyzer 104 is configured to analyze the data received from the data collector 102 in any time frame and / or for a preconfigured amount of data. As an example, the data analyzer 104 may be configured to analyze the data after a certain amount of time has passed during the data collection, or after a certain number of data points have been collected since the start of the online experiment. In one or more embodiments, the data analyzer 104 is configured to reanalyze the data from time to time. For example, if it is determined that the online experiment may not have stopped, the data analyzer 104 may be configured to reanalyze the data after a certain amount of more data has been collected. The reanalysis of the data by the data analyzer 104 may be performed at a preconfigured time interval or amount of data. In one or more embodiments, the data analyzer 104 is configured to reanalyze the data from time to time based at least in part on a previous analysis of the data (e.g., based on previously determined characteristics of the data). In one or more embodiments, after one or more characteristics of the data are determined, or when it is determined that no characteristics can be determined from the data obtained so far (e.g., when the data is random), the data analyzer 104 stops the analysis of the data. As an example, the data analyzer 104 may test the data to determine whether the data exhibits a first characteristic (e.g., the data is a constant), and when the data does not exhibit the first characteristic, test the data to determine another characteristic (e.g., it is monotonic, normally distributed, modal, autocorrelated, has a long-tailed distribution, etc.). In one or more embodiments, the data analyzer 104 is configured to determine whether the data has more than one characteristic. As an example, the data analyzer 104 may be configured to determine whether the data is also autocorrelated after determining that a data sample obtained during an online experiment is multimodal. In one or more embodiments, the data analyzer 104 is configured to provide a stopping rule applicator (discussed below) with the fact that one or more characteristics determined for the data or no characteristics were determined.

[0028] In one or more embodiments, the computing device 100 includes a stopping rule applicator 106. In one or more embodiments, the stopping rule applicator 106 is operably connected to the data analyzer 104. The stopping rule applicator 106 can be any hardware (e.g., circuitry), software, firmware, or any combination thereof configured to select a stopping rule based on one or more characteristics, or lack thereof, of the data and to apply the selected stopping rule to determine whether an appropriate convergence criterion is met.

[0029] As an example, if the data is determined (e.g., by a data analyzer) to be constant, the stopping rule selected may be to stop after a certain amount of data has been collected, the amount being preconfigured based on the type of data being collected and how much constant data should be collected to be confident that the data will not begin to change. In such an example, the convergence criterion may be whether enough data points have been obtained.

[0030] As another example, if the data is determined to have monotonic characteristics, the stopping rule can be that the online experiment can be stopped if the data has increased or decreased towards a limit and is within a threshold of the limit, or if the data has not converged to the limit, the online experiment can be stopped after a certain amount of data has been obtained but is not approaching the limit.

[0031] As another example, if the data is determined to be normally distributed, the stopping rule may be to stop the online experiment when a certain confidence interval is reached or based on the results of performing a test.

[0032] As another example, if the data is determined to be multimodal, a clustering technique may be applied to the data, and the convergence criteria may be whether the number of clusters for the best model fit is greater than 1 and whether the log-likelihood of the best model fit is below a threshold.

[0033] As another example, if the data has the property of autocorrelation, a stopping test could be that if the correlation is seen repeatedly over a certain number of time intervals, the online experiment can be stopped.

[0034] The stopping rule applicator 106 may use other stopping rules and related convergence criteria without departing from the scope of the embodiments disclosed herein (e.g., stopping based on null hypothesis testing, stopping based on Bayesian hypothesis testing, stopping based on block bootstrapping when the data is autocorrelated, stopping based on linear or Gaussian regression models, stopping based on the results of a trained machine learning model, stopping based on testing using extreme value theory, etc.).

[0035] In one or more embodiments, if the data does not have a determined characteristic, the stopping rule may be to stop the online experiment after a certain amount of time has passed or after a certain amount of data has been collected without the characteristic being determined by the data analyzer 104 .

[0036] although Figure 1 The computing device 100 is shown as including a single stopping rule applicator 106 , but the computing device 100 may include any number of stopping rule applicators without departing from the scope of the embodiments disclosed herein.

[0037] Although Figure 1Specific configurations of components are shown, but other configurations may be used without departing from the scope of the embodiments described herein. Figure 1 Some components are shown as part of the same device, but any component may be grouped into a collection of one or more components that may exist and execute as part of any number of independent and operably connected devices. As another example, a single component may be configured to perform a task performed by Figure 1 Therefore, the embodiments disclosed herein should not be limited to all or any part of the functions performed by the components shown. Figure 1 Component configuration shown.

[0038] Figure 2 An overview of an example method for automatically determining a stopping condition for an online experiment according to one or more embodiments disclosed herein is shown. Figure 2 The method shown may be performed, for example, by a computing device (e.g., Figure 1 The computing device 100 shown or Figure 3 The computing device 300 shown) and / or any component thereof (e.g., Figure 1 The data collector 102, data analyzer 104 and / or stopping rule applicator 106 are shown to perform.

[0039] Although Figure 2 The various steps in the flowcharts shown in are presented and described sequentially, but some or all of the steps may be performed in a different order, some or all of the steps may be combined or omitted, Figure 2 Other steps not shown in the figure may be performed additionally, and / or some or all steps may be performed in conjunction with Figure 2 The other steps are executed in parallel.

[0040] In step 200, the method includes obtaining a predetermined number of data points from an online experiment. In one or more embodiments, data is being obtained while the online experiment is being conducted. Discrete units of data may be referred to as data points. In one or more embodiments, data points are generated by a computing device (e.g., Figure 1 The computing device 100 shown or Figure 3 The computing device 300 shown) is a data collector (e.g., Figure 1 The data collector 102 shown in FIG. 100 can be obtained from a data source (e.g., Figure 1 In one or more embodiments, the data is obtained directly from the data source. In other embodiments, the data is obtained via a network (e.g., Figure 1In one or more embodiments, the number of data points that should be obtained before the data analysis begins is determined before the data analysis begins. In one or more embodiments, the analysis begins immediately. In other embodiments, an initial threshold number of data points (e.g., ten, one hundred, one thousand, one million, etc.) is configured to be obtained before the analysis begins. The initial threshold number of data points to be obtained before the analysis begins can be configured to any number without departing from the scope of the embodiments disclosed herein. In one or more embodiments, the initial threshold number of data points to be obtained before the analysis begins is based at least in part on the type of data to be obtained, the amount of time and / or resources to be spent on online experiments, and / or the type of results to be sought (e.g., prediction, inference, average, etc.). In one or more embodiments, the number of data points obtained before the data analysis is configurable.

[0041] In step 202, analysis of data begins. The analysis of data may be performed, for example, by a computing device (e.g., Figure 1 The computing device shown or Figure 3 The computing device 300 shown) is a data analyzer (e.g., Figure 1 204). In one or more embodiments, in step 202, the analysis includes determining whether the sample of data points obtained in step 200 has constant characteristics. In one or more embodiments, if each data point is the same (e.g., the same numerical value), the data has constant characteristics. In one or more embodiments, if each data point is within a given threshold, the data has constant characteristics. As an example, if all data points are within two percent of a value, such as all values ​​are between 98 and 102 (within one percent), the data can be determined to be constant. In one or more embodiments, if the data is determined to be constant, the method proceeds to step 214. In one or more embodiments, if the data is not determined to be constant, the method proceeds to step 204.

[0042] In step 204, the data analysis continues to determine whether the data point sample is monotonic. As used herein, monotonic refers to data having a characteristic of continuously increasing or decreasing. In one or more embodiments, monotonic data can increase or decrease toward a limit (e.g., 4, 4.5, 4.7, 4.9, 4.95, 4.98 can increase to a limit of 5), which can be referred to as converging to a limit. In one or more embodiments, monotonic data can increase or decrease indefinitely. In one or more embodiments, if the data is determined to have a monotonic characteristic, the method proceeds to step 214. In one or more embodiments, if the data is determined to not have a monotonic characteristic, the method proceeds to step 206.

[0043] In step 206, the data analysis continues to determine whether the data is normally distributed (e.g., has a Gaussian distribution). As used herein, a normal distribution can refer to data that is normally distributed, or to data for which the logarithm of the data points is normally distributed. In one or more embodiments, a normal distribution is a distribution of data around a mean, and a standard deviation of the mean can be derived. In one or more embodiments, if the data is determined to have the characteristics of a normal distribution, the method proceeds to step 214. In one or more embodiments, if the data is determined not to have the characteristics of a normal distribution, the method proceeds to step 208.

[0044] In step 208, the data analysis continues to determine whether the data has modal characteristics. The data can be unimodal but not normally distributed. For example, the data can be unimodal with a long-tailed distribution. Unimodal data can be data with a clear peak at the most frequently occurring value (e.g., the mode). In one or more embodiments, the data can be multimodal. In one or more embodiments, multimodal data as used herein is data having two or more modalities. In one or more embodiments, if the data is determined to have modal characteristics, the method continues to step 214. In one or more embodiments, if the data is determined to not have modal characteristics, the method proceeds to step 210.

[0045] In step 210, the data analysis continues to determine whether the data has the characteristic of autocorrelation. In one or more embodiments, autocorrelated data is data that has the characteristic of having a certain degree of similarity from one time period to another. For example, if the data for one day exhibits similar traffic patterns to those seen in subsequent days (e.g., daily website traffic in a particular country is low during the night and then increases during the day in the particular country), then the time series representing one day's website traffic can be determined to be autocorrelated. In one or more embodiments, if the data is determined to have the characteristic of autocorrelation, the method proceeds to step 214. In one or more embodiments, if the data is determined not to have the characteristic of autocorrelation, the method proceeds to step 212.

[0046] In step 212, it is determined whether the maximum run time of the online experiment has passed. In one or more embodiments, the maximum run time can be a configurable threshold for how long the online experiment should continue. The threshold can be an amount of time, an amount of data, or a combination thereof. The maximum run time can be configured based on any one or more of a variety of factors, which can include, but are not limited to, the amount of resources (e.g., cost, computing resources, etc.) available to conduct the online experiment; the amount of time (e.g., the results are required before a certain time, after which the results become irrelevant), etc. In one or more embodiments, if the maximum run time has passed, the method proceeds to step 216. In one or more embodiments, if the maximum run time has not passed, the method proceeds to step 214.

[0047] In step 214, it is determined whether a convergence criterion is satisfied. In one or more embodiments, as used herein, a convergence criterion is any metric that can be determined about a sample of data points that indicates that the data is capable of providing a desired result. In one or more embodiments, the convergence criterion depends at least in part on one or more characteristics of the data sample determined in steps 202-212.

[0048] As an example, if the data is determined to have constant characteristics, the convergence criterion may be a determination as to whether at least a threshold number of constant data points have been obtained. In one or more embodiments, the confidence criterion is met if a threshold number of data points has been reached.

[0049] As another example, if the data is determined to have monotonic characteristics, the convergence criteria may include determining whether the data is approaching a limit. If the monotonic data is approaching a limit, the convergence criteria may also include determining whether the data is within a certain threshold of the limit. In one or more embodiments, if the data is within the threshold, the confidence criteria are met. In one or more embodiments, if the data is determined not to have converged to a limit (e.g., it increases toward infinity), the convergence criteria may also include determining whether enough data points have been collected to ensure that the data has not increased or decreased to the limit. In one or more embodiments, if enough data points have been collected to determine that the monotonic data has not converged to the limit, the convergence criteria are met.

[0050] As another example, if the data is determined to have characteristics of a normal distribution, the convergence criteria may include applying a hypothesis test, or determining whether the width of the confidence interval for the median mean is below a threshold. In one or more embodiments, the convergence criteria are met if the mean or median of the normal distribution is within a desired confidence level.

[0051] As another example, if the data is determined to have modal characteristics, the convergence criteria may include using any of a variety of techniques applicable to unimodal or multimodal data. These techniques may include, but are not limited to, using regression models such as Gaussian mixture models, Gaussian process regression, Bayesian linear regression, kernel density estimation, etc. For such models, the convergence criteria may include determining whether the mean integrated squared error (MISE) relative to the data distribution drops below a configurable threshold, or whether the MISE does not drop below the threshold after adding a certain number of new data points after a given number of data points, both of which may indicate that the convergence criteria are met. Another technique that can be applied to modal data distributions is a clustering model, where it is determined that after a given number of samples, the number of clusters that best describe the data stops increasing, indicating that the convergence criteria are met. When the data is modal, another technique that can be applied is to use a machine learning model that is trained to determine that enough data has been obtained to be confident of the results obtained using the data obtained so far. For example, a classification model can be trained using existing and / or synthetic data to predict the category of stopping collecting data or continuing to collect data. In one or more embodiments, the convergence criteria are met if the machine learning model predicts that data collection should stop. Another technique that can be applied when the data is modal is to use extreme value theory, where the convergence criterion can include determining whether the modeled probability of observing an extreme value (e.g., an outlier) drops below a configurable threshold after a given number of samples. In one or more embodiments, the convergence criterion is met if the probability has dropped below the threshold.

[0052] Those of ordinary skill in the art who have the benefit of this disclosure will appreciate that other convergence criteria and / or other stopping rules or tests may be applied without departing from the scope of the embodiments disclosed herein. In one or more embodiments, if the convergence criteria are not met, the method proceeds to step 218. In one or more embodiments, if the convergence criteria are met, the method proceeds to step 216.

[0053] The online experiment is stopped in step 216. In one or more embodiments, the online experiment is stopped either because the data exhibits one or more characteristics corresponding to one or more stopping rules and application of those one or more stopping rules shows that convergence criteria are met, or because a maximum run time for the online experiment has elapsed.

[0054] In step 218, the method includes obtaining additional data, after which the method returns to step 202 to reanalyze the data. The amount of additional data obtained may be configurable and may be based at least in part on the data collected to date, one or more characteristics of the data obtained to date, the stopping rule and corresponding convergence criteria used in step 214, the time and resources remaining available for the online experiment, some combination of the above factors, etc.

[0055] although Figure 2 Various data tests for determining whether data has certain characteristics are shown to be performed sequentially, but in one or more embodiments, data determined to have characteristics can be tested to determine whether the data also has one or more additional characteristics. As an example, data determined to have characteristics that are modal can also be tested over time to determine whether the data is also autocorrelated. Whether to test various characteristics of the data can be configured based on domain knowledge about the type of data, the nature of the online experiment, the results sought, etc.

[0056] In one or more embodiments, obtaining additional data (e.g., as in step 218) may include determining how to process the data that has been obtained. In one or more embodiments, additional data is added to the data that has been obtained to obtain a larger data set to be analyzed for various characteristics as described above. As an example, data that was initially determined to have a constant distribution, but for which not enough data was obtained to be sure of the constant value, may be determined to be monotonic after more data is added to the data set. In one or more embodiments, at least some of the previously obtained data may be discarded. For example, the nature of the data type, the online experiment, etc. may indicate that a rolling data window obtained over a period of time should be used when testing various characteristics. As an example, the occurrence of an event during an online experiment may change the possible characteristics of the data, and therefore, it may be useful to discard the data before the event when determining the characteristics of the data. As another example, it may be determined that outliers in the data should be discarded when analyzing the data.

[0057] In one or more embodiments, although Figure 2 Stopping the online experiment as a result of meeting a convergence criterion is shown, but other techniques for making such a decision can be used. For example, if more than one characteristic is determined to correspond to the data, more than one convergence criterion can be tested. In this case, a majority voting rule can be used, where the online experiment is stopped if the convergence criteria of the majority of the tests are met, or the various stopping rules and corresponding convergence criteria can be weighted based on the observed data, or a strict strategy can be implemented, where all relevant convergence criteria must be met before the online experiment is stopped.

[0058] Figure 3A block diagram of a computing device according to one or more embodiments of the present disclosure is shown. As described above, a computing device may be used to implement the embodiments described herein. For example, one or more computing devices may be used to at least partially implement Figure 1 All or any portion of the components shown. Computing device 300 may include one or more computer processors 302, non-permanent storage 304 (e.g., volatile memory, such as random access memory (RAM), cache memory, etc.), permanent storage 306 (e.g., hard disk, optical drive such as compact disk (CD) drive or digital versatile disk (DVD) drive, flash memory, etc.), communication interface 312 (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), input device 310, output device 308, and many other elements (not shown) and functions. These components are described below.

[0059] In one or more embodiments, computer processor 302 may be an integrated circuit for processing instructions. For example, a computer processor may be one or more cores or microkernels of a processor. Processor 302 may be a general-purpose processor configured to execute program code included in software executed on computing device 300. Processor 302 may be a special-purpose processor in which certain instructions are incorporated into the processor design. Although Figure 3 Only one processor 302 is shown, but computing device 300 may include any number of processors without departing from the scope of the embodiments disclosed herein.

[0060] The computing device 300 may also include one or more input devices 310, such as a touch screen, keyboard, mouse, microphone, touchpad, electronic pen, motion sensor, or any other type of input device. The input device 310 may allow a user to interact with the computing device 300. In one or more embodiments, the computing device 300 may include one or more output devices 308, such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touch screen, a cathode ray tube (CRT) monitor, a projector, or other display device), a printer, an external memory, or any other output device. One or more output devices may be the same or different from the input device. The input and output devices may be locally or remotely connected to the computer processor 302, the non-permanent storage 304, and the permanent storage 306. There are many different types of computing devices, and the above-mentioned input and output devices may take other forms. In some cases, a multimodal system may allow a user to provide multiple types of input / output to communicate with the computing device 300.

[0061] Additionally, communication interface 312 may facilitate connecting computing device 300 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or to another device such as another computing device. Communication interface 312 may use wired and / or wireless transceivers to perform or facilitate the reception and / or transmission of wired or wireless communications, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Proprietary wired ports / plugs, Wireless signal transmission, BLE wireless signal transmission, Wireless signal transmission, RFID wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 WiFi wireless signal transmission, WLAN signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 312 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing device 300 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, GPS of the United States, Global Navigation Satellite System (GLONASS) of Russia, Beidou Navigation Satellite System (BDS) of China, and Galileo GNSS of Europe. There is no restriction as to operation on any particular hardware configuration, so the basic features herein may be readily replaced with improved hardware or firmware configurations as they evolve.

[0062] The term computer-readable medium includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored, and does not include carrier waves and / or temporary electronic signals that are wirelessly or propagated through wired connections. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media such as CDs or DVDs, flash memory, memory, or memory devices. Computer-readable media may store codes and / or machine-executable instructions thereon, which may represent any combination of processes, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0063] All or any portion of the components of computing device 300 may be implemented in circuits. For example, a component may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, GPU, DSP, CPU, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. In some aspects, computer-readable storage devices, media, and memories may include wired or wireless signals including bitstreams, etc. However, when referred to, non-transitory computer-readable storage media expressly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0064] In the above description, many details are set forth as examples of the embodiments described herein. Those skilled in the art (who also benefit from this disclosure) will appreciate that one or more of the embodiments described herein may be implemented without these specific details, and that many variations or modifications are possible without departing from the scope of the embodiments described herein. Certain details known to those of ordinary skill in the art may be omitted to avoid obscuring the description.

[0065] Specific details are provided in the above description to provide a comprehensive understanding of the aspects and examples provided herein. However, it will be understood by those of ordinary skill in the art that these aspects can be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including functional blocks, which may include devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. In addition to the components shown in the figures and / or described herein, other components may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring these aspects with unnecessary details. In other cases, in order to avoid obscuring these aspects, known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details.

[0066] Various aspects may be described above as processes or methods shown in flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flow charts may describe operations as sequential processes, many operations may be performed in parallel or concurrently. In addition, the order of operations may be rearranged. When the operations of a process are completed, the process is terminated, but there may be additional steps not included in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or main function.

[0067] The process and method according to the above examples can be implemented using computer executable instructions stored in or otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or function group. Some of the computer resources used may be accessed through a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information, and / or information created during the method according to the examples include disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0068] In the above description of the drawings, in the various embodiments described herein, any component described with respect to a drawing may be equivalent to one or more identically or similarly named and / or numbered components described with respect to any other drawing. For the sake of brevity, the description of these components may not be repeated for each drawing. Therefore, each embodiment of the components of each drawing is incorporated by reference and is assumed to be optionally present in each other drawing with one or more identically or similarly named and / or numbered components. In addition, according to the various embodiments described herein, any description of the drawing components should be interpreted as an optional embodiment, which can be implemented as a supplement, combination or replacement of the embodiments described with respect to the corresponding one or more identically or similarly named and / or numbered components in any other drawing.

[0069] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives of elements (i.e., any nouns in the application). The use of ordinal numbers does not imply or create any particular order of elements, nor does it limit any element to only a single element, unless explicitly disclosed, such as by the use of the terms "before," "after," "single," and other such terms. Instead, ordinal numbers are used to distinguish elements. For example, a first element is different from a second element, and a first element may contain more than one element and be after (or before) a second element in the ordering of elements.

[0070] As used herein, the phrase operably connected or operably connected means that there is a direct or indirect connection between elements / components / devices that allows the elements to interact with each other in some manner. For example, the phrase "operably connected" can refer to any direct (e.g., a wired connection directly between two devices or components) or indirect (e.g., a wired and / or wireless connection between any number of devices or components that connect operably connected devices) connection. Therefore, any path that information can pass can be considered an effective connection.

[0071] Embodiments disclosed herein describe systems and methods for automating a stopping decision process for online experiments. The disclosed stopping decision process includes obtaining data in real time during an online experiment, dynamically analyzing the data to determine whether the data exhibits one or more characteristics, selecting a stopping rule to be tested and a corresponding convergence criterion based on the one or more characteristics of the data, and stopping the online experiment when it is determined that the convergence criterion is met. In one or more embodiments, the automatic stopping decision process can dynamically react to changes in the data and allow sufficient data to be collected to be confident of the results obtained based on the data, and also allow data collection to be stopped, thereby avoiding unnecessary waste of time or other resources used during online experiment data collection.

[0072] Although the embodiments discussed herein have been described with respect to a limited number of embodiments, those skilled in the art having benefit of this disclosure will appreciate that other embodiments may be designed without departing from the scope of the embodiments disclosed herein. Accordingly, the scope of the embodiments described herein should be limited solely by the appended claims.

Claims

1. A device comprising: one or more processors; as well as One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to: obtaining a predetermined number of data points from an online experiment; processing the data points to determine whether the data points exhibit at least one characteristic; In response to the data point exhibiting the at least one characteristic, selecting a first stopping rule from a plurality of stopping rules, wherein the first stopping rule corresponds to the at least one characteristic; applying the first stopping rule to determine whether a first convergence criterion is met; and When the first convergence criterion is met, the online experiment is stopped.

2. The device according to claim 1, wherein: When the first convergence criterion is not met, execution of the instructions further causes the one or more processors to: Get additional data points.

3. The device according to claim 2, wherein: Execution of the instructions further causes the one or more processors, after obtaining the additional data point: determining that the data point and the additional data point do not exhibit a characteristic; and The online experiment was stopped after a maximum run time had elapsed.

4. The device according to claim 1, wherein: Execution of the instructions further causes the one or more processors to, before stopping the online experiment: In response to the data point exhibiting a second characteristic, selecting a second stopping rule from the plurality of stopping rules, wherein the second stopping rule corresponds to the second characteristic; as well as applying the second stopping rule to determine whether a second convergence criterion is met, Wherein, when the first convergence criterion and the second convergence criterion are met, the online experiment is stopped.

5. The device according to claim 1, wherein: The at least one characteristic includes that the data points are constant, and the first convergence criterion includes determining whether a number of the data points exceeds a threshold.

6. The device according to claim 1, wherein: The at least one characteristic includes that the data points are monotonic, and the first convergence criterion includes determining that the data points are within a threshold of a limit.

7. The device according to claim 1, wherein: The at least one characteristic includes that the data points have a normal distribution, and the first convergence criterion includes determining that a mean of the data points is within a confidence interval of a specified maximum width.

8. The device according to claim 1, wherein: The at least one characteristic includes that the data points are modal, and the first convergence criterion includes applying a regression model or a clustering model.

9. The device according to claim 1, wherein: The at least one characteristic includes that the data points are autocorrelated.

10. A computer-implemented method comprising: obtaining, by a processing device, a predetermined number of data points from an online experiment; processing the data points by the processing device to determine whether the data points exhibit at least one characteristic; selecting, by the processing device, a first stopping rule from a plurality of stopping rules in response to the data point exhibiting the at least one characteristic, wherein the first stopping rule corresponds to the at least one characteristic; applying, by the processing device, the first stopping rule to determine whether a first convergence criterion is satisfied; and When the first convergence criterion is met, the online experiment is stopped.

11. The method according to claim 10, further comprising, when the first convergence criterion is not met: Get additional data points.

12. The method of claim 11, further comprising, after obtaining the additional data point: determining that the data point and the additional data point do not exhibit a characteristic; and The online experiment was stopped after a maximum run time had elapsed.

13. The method according to claim 10, further comprising, before stopping the online experiment: In response to the data point exhibiting a second characteristic, selecting a second stopping rule from the plurality of stopping rules, wherein the second stopping rule corresponds to the second characteristic; and applying the second stopping rule to determine whether a second convergence criterion is met, in, When the first convergence criterion and the second convergence criterion are met, the online experiment is stopped.

14. The method according to claim 10, wherein: The at least one characteristic includes that the data points are constant, and the first convergence criterion includes determining whether a number of the data points exceeds a threshold.

15. The method according to claim 10, wherein: The at least one characteristic includes that the data points are monotonic, and the first convergence criterion includes determining that the data points are within a threshold of a limit.

16. The method according to claim 10, wherein: The at least one characteristic includes that the data points have a normal distribution, and the first convergence criterion includes determining that a mean of the data points is within a confidence interval of a specified maximum width.

17. The method according to claim 10, wherein: The at least one characteristic includes that the data points are modal, and the first convergence criterion includes applying a regression model or a clustering model.

18. The method according to claim 10, wherein: The at least one characteristic includes that the data points are autocorrelated.

19. A non-transitory computer-readable medium storing a program for execution by one or more processors, the program comprising instructions for: obtaining, by a processing device, a predetermined number of data points from an online experiment; processing the data points by the processing device to determine whether the data points exhibit at least one characteristic; selecting, by the processing device, a first stopping rule from a plurality of stopping rules in response to the data point exhibiting the at least one characteristic, wherein the first stopping rule corresponds to the at least one characteristic; applying, by the processing device, the first stopping rule to determine whether a first convergence criterion is satisfied; and When the first convergence criterion is met, the online experiment is stopped.

20. The non-transitory computer readable medium of claim 19, wherein: The program also includes instructions for obtaining additional data points when the first convergence criterion is not met.

Images