Systems and methods for large scale variant testing

Abstract

Techniques for large scale variant testing are presented. In particular, the set forth embodiments provide systems and methods for testing, on a large scale software application store, visual aspects of one or more variants of representative data associated with applications available through the software application store. According to some embodiments, a method may include calculating at least one transformation metric for a control object and at least one transformation metric for at least one variant object using a subset of transformation data associated with the control object and a subset of the transformation data associated with the at least one variant object. The method may also include generating a performance measurement by applying at least one statistical hypothesis test function to the at least one conversion metric of the control object and the at least one conversion metric of the at least one variant object.

Description

Technical Field

[0001] The described embodiments set forth techniques for large-scale variant testing. Specifically, the described embodiments provide systems and methods for testing visual aspects of one or more variants of representative data associated with an application available on a large-scale software application store. Background Art

[0002] In recent years, downloading software applications (or "apps") from software application stores has become a popular method for obtaining software applications. Software application stores ("app stores") allow users to download software applications ("apps") to their devices (such as desktop or laptop computers, smart phones, etc.), and then install the applications on their devices. Before downloading an application, users often browse for applications within the app store. For example, in response to a user search, the app store may provide the user with results having a particular set of representative data, such as icons, screenshots, text descriptions, etc.

[0003] An application developer may periodically test one or more variations of such representative data. For example, an application developer may modify an icon, screenshot, etc. of a page associated with an application in an attempt to increase conversion rates (e.g., the rate of users visiting the page and the number of downloads of the application). In this way, an application developer may test multiple variations to determine which variation results in the highest conversion rate. For large-scale software application stores (e.g., with potentially millions of application developers), such testing may be very resource intensive when performed by any number of application developers and may take a relatively long time to provide variation test results to the application developers. Summary of the invention

[0004] The present application proposes techniques for large-scale variant testing. Specifically, the described embodiments provide systems and methods for testing visual aspects of one or more variants of representative data associated with applications available on a large-scale software application store.

[0005] One embodiment describes a method for scalable variant testing, the method comprising receiving a variant test input indicating user information and at least one variant feature for a variant test associated with a control feature. The method also comprises identifying a control object corresponding to the control feature associated with the user information based on the user information, and generating at least one variant object based on the at least one variant feature. The method also comprises collecting transformation data of the control object and the transformation data of the at least one variant object, storing the transformation data of the control object and the transformation data of the at least one variant object in a fact database according to a collection date, and retrieving a subset of the transformation data associated with the control object and a subset of the transformation data associated with the at least one variant object from the fact database. The method also comprises calculating at least one transformation metric of the control object using the subset of the transformation data associated with the control object, calculating at least one transformation metric of the at least one variant object using the subset of the transformation data associated with the control object, generating a performance measure by applying at least one statistical hypothesis testing function to the at least one transformation metric of the control object and the at least one transformation metric of the at least one variant object, and providing the performance measure at a display.

[0006] Other embodiments include a non-transitory computer-readable storage medium configured to store instructions that, when executed by a processor included in a computing device, cause the computing device to implement the methods and techniques described in the present disclosure. Still other embodiments include a hardware computing device that includes a processor that can be configured to cause the hardware computing device to implement the methods and techniques described in the present disclosure.

[0007] Other aspects and advantages of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings which illustrate by way of example the principles of the described embodiments.

[0008] The present disclosure is provided for the purpose of summarizing some example embodiments only, so as to provide a basic understanding of some aspects of the subject matter described herein. Therefore, it should be understood that the above-mentioned features are only examples and should not be construed as narrowing the scope or essence of the subject matter described herein in any way. Other features, aspects and advantages of the subject matter described herein will become apparent through the following detailed description, drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present disclosure will be more readily understood through the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals designate like structural elements.

[0010] Figure 1 A conceptual diagram illustrating different components of a computing device configured to implement the various techniques described herein, according to some embodiments.

[0011] Figure 2 A block diagram of a scalable variant testing system is illustrated according to some embodiments.

[0012] Figure 3A and Figure 3B A block diagram of an alternative scalable variant testing system is illustrated in accordance with some embodiments.

[0013] Figure 4A and Figure 4B A scalable variant testing method according to some embodiments is illustrated.

[0014] Figure 5 A block diagram illustrating exemplary elements of a mobile wireless device according to some embodiments is illustrated. DETAILED DESCRIPTION

[0015] Representative applications of the methods and apparatus according to the present application are described in this section. These examples are provided only to add context and aid in understanding the described embodiments. Therefore, it will be apparent to those skilled in the art that the described embodiments may be practiced without some or all of these specific details. In other cases, in order to avoid unnecessarily obscuring the described embodiments, well-known processing steps are not described in detail. Other applications are possible, so that the following examples should not be considered limiting.

[0016] In the following detailed description, reference is made to the accompanying drawings which form a part of the specification and in which are shown by way of illustration specific embodiments in accordance with the described embodiments. Although these embodiments are described in sufficient detail to enable those skilled in the art to practice the described embodiments, it is to be understood that these examples are not limiting; other embodiments may be used and modifications may be made without departing from the spirit and scope of the described embodiments.

[0017] The embodiments described herein set forth techniques for enabling large-scale variant testing to be performed in an efficient manner. In some embodiments, the systems and methods described herein may be configured to provide conversion rate estimates by using statistical regularization to more accurately estimate the conversion rate of a variant in an experiment based on the signal-to-noise ratio observed in the data (e.g., relative to natural estimates of conversions and / or impressions). The systems and methods described herein may be configured to provide confidence estimates by dynamically defining hypothesis tests that evolve over the course of a single experiment in response to data observed during the experiment.

[0018] According to some embodiments, the systems and methods described herein may be configured to provide invalidity estimates by using Monte Carlo simulations to robustly predict the tendency of a test to be invalid (e.g., an estimate that no significant results will be obtained within a period of time (such as 90 days or other suitable period)). The systems and methods described herein may be configured to provide a duration calculator using machine learning models and Monte Carlo simulations to provide fast Bayesian power analysis. The systems and methods described herein may be configured to provide a data engineering method that provides a single pass for processing all running experiments with pluggable statistical methods.

[0019] In some embodiments, the systems and methods described herein may be configured to provide an experimental platform that provides daily user profiles, aggregates metrics for the daily user profiles over a run cycle, and applies statistical methods to the aggregated metrics. The systems and methods described herein may be configured to aggregate metrics at each user level and verify experimental hypotheses in response to an experiment with users as randomization units. The systems and methods described herein may be configured to aggregate the mean of each metric for each variant using a two-sample T-test.

[0020] In some embodiments, the systems and methods described herein can be configured to scan aggregated metrics multiple times without repeatedly processing user profiles. The systems and methods described herein can be configured so that when multiple experiments are performed simultaneously or substantially simultaneously, each experiment initiates a single query that can trigger the same data scan of daily user profiles multiple times.

[0021] In some embodiments, the systems and methods described herein may be configured to receive a variant test input indicating user information and at least one variant feature for a variant test associated with a control feature. The systems and methods described herein may be configured to identify a control object corresponding to the control feature associated with the user information based on the user information. The systems and methods described herein may be configured to generate at least one variant object based on the at least one variant feature.

[0022] The systems and methods described herein may be configured to collect conversion data for a control object and conversion data for the at least one variant object. The conversion data for the control object may be generated based on the number of unique users that accessed the control object, and for each respective user associated with the number of unique users that accessed the control object, one of the following: a first value indicates that the respective user downloaded an application associated with the control object when the respective user first accessed the control object, and a second value indicates that the respective user did not download an application associated with the control object when the respective user first accessed the control object. The conversion data for the at least one variant object may be generated based on the number of unique users that accessed the variant object, and for each respective user associated with the number of unique users that accessed the variant object, one of the following: a first value indicates that the respective user downloaded an application associated with the variant object when the respective user first accessed the variant object, and a second value indicates that the respective user did not download an application associated with the variant object when the respective user first accessed the variant object.

[0023] The systems and methods described herein may be configured to store the conversion data of the control object and the conversion data of the at least one variant object in a fact database according to a collection date. The systems and methods described herein may be configured to retrieve a subset of the conversion data associated with the control object and a subset of the conversion data associated with the at least one variant object from the fact database. The subset of the conversion data associated with the control object and the subset of the conversion data associated with the at least one variant object may correspond to a collection period. The collection period may include any suitable collection period, such as a day, a week, a month, etc.

[0024] The systems and methods described herein may be configured to calculate at least one conversion metric of the control object using a subset of conversion data associated with the control object. The systems and methods described herein may be configured to calculate at least one conversion metric of the at least one variant object using a subset of conversion data associated with the control object. The systems and methods described herein may be configured to generate performance measurements of variant tests by applying at least one statistical hypothesis testing function to the at least one conversion metric of the control object and the at least one conversion metric of the at least one variant object. Performance measurements may be generated within a collection cycle and / or any suitable cycle. The at least one statistical hypothesis testing function may include a Bayesian sequential test function, a chi-square test function, any other suitable statistical hypothesis testing function, or a combination thereof. The systems and methods described herein may be configured to provide performance measurements at a display.

[0025] These and other embodiments are referred to below. Figures 1 to 5 Discussion is made; however, those skilled in the art will readily appreciate that the detailed description given herein with respect to these figures is for illustrative purposes only and should not be construed as limiting.

[0026] Figure 1 1 illustrates a conceptual diagram of a computing device 102 (e.g., a smartphone, a tablet computer, a laptop computer, a desktop computer, a server, etc.) that can be configured to implement the various techniques described herein. Figure 1 As shown, the computing device 102 may include a processor 104 that, in conjunction with a volatile memory 106 (e.g., dynamic random access memory (DRAM)) and a storage device 112 (e.g., a solid-state drive (SSD)), enables execution of different software entities on the computing device 102. For example, the processor 104 may be configured to load various components of an operating system (OS) 108 from the storage device 112 into the volatile memory 106. In turn, the operating system 108 may enable the computing device 102 to provide various useful functions, such as loading / executing various applications 110 (e.g., user applications). It should be understood that for simplicity, Figure 1 The various hardware components of the illustrated computing device 102 are presented at a high level and are described below in conjunction with Figure 5 Provides a more detailed breakdown.

[0027] like Figure 1 As shown, the operating system 108 / application 110 may issue a write command 130 to the storage device 112, for example, to write new data, overwrite existing data, migrate existing data, etc. According to some embodiments, and as Figure 1 As shown, the storage device 112 may include a controller 114 configured to coordinate the overall operation of the storage device 112. Specifically, the controller 114 may implement a write cache manager 116 that receives various write commands 130 and stores them in a write cache 118. The write cache manager 116 may send the write commands 130 stored in the write cache 118 to the non-volatile memory 120. According to some embodiments, the non-volatile memory 120 may include log information configured to receive transaction information associated with details associated with I / O requests processed by the controller 114.

[0028] In some embodiments, computing device 102 may be configured to provide scalable variant testing. Scalable variant testing may be associated with variant testing on a large-scale software application store and / or any other suitable platform or application. Computing device 102 may be configured to provide a dedicated analysis platform to support monitoring testing of, for example, product pages (e.g., pages on an application store associated with a downloadable application). It should be noted that while the systems and methods described herein are generally described in the context of a large-scale software application store, the systems and methods described herein may be applied to any suitable platform, application, etc. that utilizes variant testing.

[0029] refer to Figure 2 , generally illustrating an extensible variant testing system 200. System 200 may enable an application developer to manage experiments with product page optimizations and publish such product pages. A client device may interact with experiments on an application store via an application store connection application 202. An application developer may create, configure, stop, and / or publish an application on an application store via the application store connection application 202. An application developer may create one or more experiments for one or more variants of a visual aspect of a product page. For example, an application developer may use an experiment management application 204 associated with the application store connection application 202. An application developer may create experiments, configure experiments, and / or apply treatments (e.g., which may be referred to herein as variants corresponding to changes in the visual aspect of a product page). Computing device 102 may apply one or more variants to a corresponding application of the application developer on application store 206. For example, computing device 102 may assign a first variant to a first version of a product page, a second variant to a second version of a product page, and so on (e.g., depending on the number of variants being tested). Computing device 102 may configure each version of the product page so that each version of the product page is accessed by a corresponding end user (eg, a user accessing the product page to potentially download an application associated with the product page).

[0030] In some embodiments, when an end user accesses the application store 206, the end user may navigate to a product page. The end user may be presented with one of the control versions of the product page (e.g., which may correspond to an unaltered or original version of the product page or other suitable control versions of the product page) or one of the variant versions of the product page. The end user may download an application associated with the product page, or may not download an application associated with the product page. The computing device 102 may store conversion data associated with each of the control version of the product page and the variant of the product page visited by the end user in the log database 208. The conversion data may include a date of visit, a user identifier, a product page identifier (e.g., indicating which versions of the product page the end user visited), and a first value that is set if the end user downloaded an application associated with the product page, a second value that is set if the end user did not download an application associated with the product page, and / or any other suitable data.

[0031] In some embodiments, computing device 102 may use various experimental analysis pipelines 210 to update the performance of variant experiments. For example, experimental analysis pipeline 210 may update performance data daily or according to any suitable period. Computing device 102 may store the performance data in an experimental performance results database 212. Application analysis application 214 may use the performance results to monitor various variant experiments. Computing device 102 may provide the performance results at a display. Application developers may view the performance results at the display.

[0032] Product page conversion rate (e.g., the ratio of users who visit an app store 206 product page and download the app associated with the product page) is a key metric for product page apps. Even a very small improvement in product page conversion rate can make a big difference in the growth of the app audience. The goal of product page optimization is to allow developers to experiment with their product pages and make data-driven decisions. When observing different conversion rates between product page variations, developers can measure how meaningful the results will be through statistical tests.

[0033] Therefore, computing device 102 can use many statistical tests to evaluate conversion data. For example, computing device 102 can use chi-square test to analyze conversion data to evaluate the change of conversion data. The performance results determined using chi-square test can be generated (and visible to application developers) from the day after the test starts, which allows application developers to view metrics every day. In order to avoid increasing the error rate, computing device 102 can use Bayesian factors to quantify relative evidence. In this regard, computing device 102 can be configured to support the application of various statistical methods without modifying the entire data pipeline and recalculating all data sets. Additionally or alternatively, computing device 102 can be configured to use per-unit statistics to measure the metrics of each variant, to build a single job to aggregate the metrics of all experiments, and / or to use one or more programming languages ​​to process large-scale data.

[0034] In some embodiments, the computing device 102 may be configured to collect raw instrumentation data to measure metrics for each variant. The computing device 102 may collect relevant logs to derive metrics for each randomization unit, which constitute the "what" or "who" of the variants assigned to the experiment. The collected raw instrumentation data may be referred to as "facts" herein. The computing device 102 may store fact data based on randomization units in a table to scale metric aggregation (e.g., this may help avoid scanning a relatively large amount of raw instrumentation data multiple times).

[0035] To measure the conversion rate of product page optimization, computing device 102 may collect: which product page variants users have visited; whether the product page is a control or a treatment (e.g., a variant product page); and whether those visits have resulted in the user downloading the app (e.g., using the first and second values ​​described herein, true or false indicators, etc.). Computing device 102 may protect user privacy by not storing personally identifiable information, while only maintaining distinct and consistent values ​​throughout the experiment. Because data processing occurs daily, computing device 102 may partition fact data by date in order to prune old data.

[0036] It would be difficult and time-consuming to derive the above-mentioned patterns from the raw data, which would require sifting through hundreds of columns of data. In addition, retrieving the experiment identification information and variant identification information for a particular visitor may be significantly more difficult than retrieving the columns from the raw data, because retrieving the correct data may depend on the system configuration mode of the experiment and the variant allocation service. Therefore, the computing device 102 may collect this more complex upstream instrumentation data to help scale further processing (e.g., which may include scanning multiple data sources and applying various types of join / sort logic to obtain the final daily fact data).

[0037] Figure 3A An alternative scalable variant testing system 300 is generally illustrated. Specifically, computing device 102 can be configured to use system 300 to provide a single-pass metric calculation. For example, computing device 102 can be configured to process all metrics for all active experiments in a single-pass data pipeline. Computing device 102 can calculate metrics for each variant of an active experiment. For example, computing device 102 can use data 302 to perform a daily fact aggregation job 304, which can include data for visiting product pages, data for downloading from product pages, and / or data for other metrics. Fact aggregation job 304 can perform concatenation of visitor identification information, enrichment of data 302 with variant allocation information, etc.

[0038] As described, computing device 102 stores fact data by date in fact database 306. Computing device 102 may use the fact data as input for statistical hypothesis testing. Depending on the statistical method and the primary metric of the experiment, the calculated data may be formatted accordingly. For example, a conversion rate test may include two sufficient statistics: the number of unique visitors who downloaded an app and the number of visitors who did not download the app. Computing device 102 may be configured to calculate metrics using any suitable query language. Computing device 102 may run a single query for each experiment.

[0039] In some embodiments, computing device 102 may persist intermediate data rather than simply querying, and may join the experiment metadata table to retrieve necessary information, including the start date. For example, computing device 102 may perform metric calculation job 308, which may include filtering all data partitions older than the earliest start date of all running experiments, joining experiment metadata 310 to experiment identification information and status data, etc.

[0040] The computing device 102 may retrieve the earliest start date of the running experiment from the fact database 306 to delete the data partition that does not contain any relevant events. For the date partitions ranging from the earliest start date to the current processing date, the computing device 102 may connect the experiment metadata 310 with facts indicating whether the product page was visited and whether the application was downloaded, which may correspond to the experiment identification information and / or the start date.

[0041] The computing device 102 may sort the facts indicating whether the product page is visited and whether the application is downloaded by the experiment identification information, which may be stored as a first sorting column to improve the performance of connecting the experiment identification information. The computing device 102 may be configured to use a common table expression written as a function to be unit tested. The computing device 102 may cache or persist intermediate data to avoid recalculating such data. Additionally or alternatively, the computing device 102 may be configured to calculate multiple metrics 312 in parallel.

[0042] In some embodiments, the computing device 102 may be configured to provide intermediate data materialization. The computing device 102 may perform an aggregation job for all experiments. The computing device 102 may use the results of the metrics 312 to perform one or more statistical hypothesis testing jobs 314. For example, the computing device 102 may use a Bayesian sequential test of conversion rates by comparing the same metrics of the metrics 312 that evolve daily from the first day of the experiment. If the dependency between the metric implementation and the upstream input data can be decomposed by having a per-unit fact table, then updating or adding a metric definition can be accomplished with minimal changes to the pipeline implementation.

[0043] like Figure 3B As illustrated in conceptual diagram 350 of , computing device 102 may be configured to simplify statistical hypothesis testing to: reading metric data; grouping by experiment identification information element; and applying user defined functions (UDFs) via an application programming interface (API). Computing device 102 may be configured to reference experimental metadata, such as the number of variants, start date, and significance level, when running one or more statistical calculations. Computing device 102 may be configured to perform statistical hypothesis testing on a corresponding experiment (e.g., in Figure 3BThe data frame data (indicated as Experiment 1, Experiment 2, etc. in the example) is enriched with the required elements before being passed to the UDF. The UDF can be configured as a shared library, and the library can be integrated into the post-processing task.

[0044] In some embodiments, computing device 102 may aggregate randomized unit-level metrics. For product page optimization, computing device 102 may provide visit and download information per user, indicating the variants in use. Computing device 102 may partition the collection of per-unit statistics by date to support tracking experimental performance results and pruning obsolete data, which may enable more flexibility in metric implementation and may allow for the use of various statistical methods for hypothesis testing.

[0045] In some embodiments, computing device 102 can scale pipelines with small computing task units instead of many fine-grained tasks, which can provide improved scaling and can prevent bottlenecks in distributed computing environments, such as query execution resource planning, repeatedly reading the same instrument data, and excessive filtering. In some embodiments, computing device 102 can use scalable programming languages ​​and UDFs to bridge the gap between data science and engineering without performance issues.

[0046] In some embodiments, computing device 102 may receive a variant test input indicating user information and at least one variant feature for a variant test associated with a control feature. Computing device 102 may identify a control object corresponding to the control feature associated with the user information based on the user information. Computing device 102 may generate at least one variant object based on the at least one variant feature.

[0047] The computing device 102 may collect conversion data for the control object and conversion data for the at least one variant object. The conversion data for the control object may be generated based on the number of unique users that accessed the control object, and for each respective user associated with the number of unique users that accessed the control object, one of the following: a first value indicating that the respective user downloaded an application associated with the control object when the respective user first accessed the control object, and a second value indicating that the respective user did not download an application associated with the control object when the respective user first accessed the control object. The conversion data for the at least one variant object may be generated based on the number of unique users that accessed the variant object, and for each respective user associated with the number of unique users that accessed the variant object, one of the following: a first value indicating that the respective user downloaded an application associated with the variant object when the respective user first accessed the variant object, and a second value indicating that the respective user did not download an application associated with the variant object when the respective user first accessed the variant object. The conversion data may be stored in the fact database 306.

[0048] The computing device 102 may retrieve a subset of the transformation data associated with the control object and a subset of the transformation data associated with the at least one variant object from the fact database 306. The subset of the transformation data associated with the control object and the subset of the transformation data associated with the at least one variant object may correspond to a collection period. The collection period may include any suitable collection period, such as a day, a week, a month, etc.

[0049] The systems and methods described herein may be configured to calculate at least one conversion metric of the control object using a subset of conversion data associated with the control object. The computing device 102 may calculate at least one conversion metric of the at least one variant object using a subset of conversion data associated with the control object. The computing device 102 may store each conversion metric in the metric 312. The computing device 102 may generate a performance measure of the variant test by applying at least one statistical hypothesis testing function (e.g., associated with the statistical hypothesis testing job 314) to the at least one conversion metric of the control object and the at least one conversion metric of the at least one variant object. The performance measure may be generated within a collection period and / or any suitable period. The at least one statistical hypothesis testing function may include a Bayesian sequential test function, a chi-square test function, any other suitable statistical hypothesis testing function, or a combination thereof.

[0050] Computing device 102 may store the performance results in experiment performance results database 316. Computing device 102 may provide the performance measurements at a display (eg, including any suitable display such as those described herein).

[0051] In some embodiments, the computing device 102 may perform the methods described herein. However, the methods described herein performed by the computing device 102 are not intended to be limiting, and any type of software executed on a processor may perform the methods described herein without departing from the scope of the present disclosure. For example, a processor executing software within another computing device may perform the methods described herein.

[0052] Figure 4A and Figure 4B An extensible variant testing method 400 is illustrated according to some embodiments. At step 402, the method 400 receives a variant test input indicating user information and at least one variant feature for a variant test associated with a control feature.

[0053] At step 404, the method 400 identifies a control object corresponding to a control feature associated with the user information based on the user information. At step 406, the method 400 generates at least one variant object based on the at least one variant feature. At step 408, the method 400 collects conversion data of the control object and conversion data of the at least one variant object. At step 410, the method 400 stores the conversion data of the control object and the conversion data of the at least one variant object in a fact database according to a collection date. At step 412, the method 400 retrieves a subset of conversion data associated with the control object and a subset of conversion data associated with the at least one variant object from the fact database.

[0054] Now go to Figure 4B At step 414, the method 400 calculates at least one conversion metric for the control object using the conversion data subset associated with the control object. At step 416, the method calculates at least one conversion metric for the at least one variant object using the conversion data subset associated with the control object. At step 418, the method 400 generates a performance measure for the variant test by applying at least one statistical hypothesis testing function to the at least one conversion metric for the control object and the at least one conversion metric for the at least one variant object. At step 420, the method 400 provides the performance measure at a display.

[0055] Figure 5 1 illustrates a detailed view of a representative computing device 500 that can be used to implement the various methods described herein according to some embodiments. In particular, the detailed view illustrates various components that can be included in the computing device 102. Figure 5 As shown, the computing device 500 may include a processor 502, which represents a microprocessor or controller for controlling the overall operation of the computing device 500. The computing device 500 may also include a user input device 508 that allows a user of the computing device 500 to interact with the computing device 500. For example, the user input device 508 may take a variety of forms, such as buttons, keypads, dials, touch screens, audio input interfaces, visual / image capture input interfaces, input in the form of sensor data, etc. Further, the computing device 500 may include a display 510 that can be controlled by the processor 502 to display information to the user. The data bus 516 may facilitate data transmission between at least the storage device 540, the processor 502, and the controller 513. The controller 513 can be used to interact with and control different equipment through the equipment control bus 514. The computing device 500 may also include a network / bus interface 511 that is communicatively coupled to the data link 512. In the case of a wireless connection, the network / bus interface 511 may include a wireless transceiver.

[0056] The computing device 500 also includes a storage device 540, which may include a single disk or multiple disks (e.g., a hard drive), and includes a storage management module that manages one or more partitions within the storage device 540. In some embodiments, the storage device 540 may include flash memory, semiconductor (solid-state) memory, etc. The computing device 500 may also include a random access memory (RAM) 520 and a read-only memory (ROM) 522. The ROM 522 may store programs, utilities, or processes to be executed in a non-volatile manner. The RAM 520 may provide volatile data storage and store instructions related to the operation of the computing device 500. The computing device 500 may also include a secure element (SE) 524 for cellular wireless system access by the computing device 500.

[0057] The various aspects, embodiments, specific implementations or features of the described embodiments may be used individually or in any combination. Various aspects of the described embodiments may be implemented by software, hardware, or a combination of hardware and software. The described embodiments may also be implemented as computer-readable code on a non-transient computer-readable medium. A non-transient computer-readable medium is any data storage device that can store data, which can then be read by a computer system. Examples of non-transient computer-readable media include read-only memory, random access memory, CD-ROM, HDD, DVD, magnetic tape, and optical data storage devices. Non-transient computer-readable media may also be distributed on network-coupled computer systems so that the computer-readable code is stored and executed in a distributed manner.

[0058] In connection with this disclosure, it is understood that the use of personally identifiable information should be subject to privacy policies and practices that are generally recognized to meet or exceed industry or government requirements for maintaining user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of the authorized use should be clearly stated to users.

[0059] For the purpose of explanation, the foregoing description uses specific nomenclature to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that specific details are not required in order to practice the described embodiments. Therefore, the foregoing description of specific embodiments is presented for the purpose of illustration and description. The foregoing description is not intended to be exhaustive or to limit the described embodiments to the precise form disclosed. It will be apparent to those of ordinary skill in the art that, in view of the above teachings, many modifications and variations are possible.

Claims

1. A method for scalable variant testing, the method comprising: receiving, for a variant test associated with a control feature, a variant test input indicating user information and at least one variant feature; identifying, based on the user information, a control object corresponding to a control feature associated with the user information; generating at least one variant object based on the at least one variant feature; collecting transformation data of the control object and transformation data of the at least one variant object; storing the transformation data of the control object and the transformation data of the at least one variant object in a fact database according to a collection date; retrieving from the fact database a subset of transformation data associated with the control object and a subset of the transformation data associated with the at least one variant object; calculating at least one conversion metric for the control object using the subset of conversion data associated with the control object; calculating at least one transformation metric for the at least one variant object using the subset of transformation data associated with the control object; generating a performance measure for the variant test by applying at least one statistical hypothesis testing function to the at least one transformation metric for the control subject and the at least one transformation metric for the at least one variant subject; as well as The performance measurements are provided at a display.

2. The method according to claim 1, wherein: generating the conversion data for the control object based on the number of unique users accessing the control object, and For each corresponding user associated with the number of unique users accessing the control object, one of the following: a first value indicating that the corresponding user downloaded an application associated with the control object when the corresponding user first accessed the control object, and a second value indicating that the corresponding user did not download the application associated with the control object when the corresponding user first accessed the control object.

3. The method according to claim 1, wherein: generating the conversion data for the at least one variant object based on a number of unique users accessing the at least one variant object, and For each corresponding user associated with the number of unique users accessing the at least one variant object, one of: a first value indicating that the corresponding user downloaded an application associated with the at least one variant object when the corresponding user first accessed the at least one variant object, and a second value indicating that the corresponding user did not download the application associated with the at least one variant object when the corresponding user first accessed the at least one variant object. 4 . The method of claim 1 , wherein the subset of transformation data associated with the control object and the subset of transformation data associated with the at least one variant object correspond to a collection cycle. The method of claim 4 , wherein the collection period comprises one day. The method of claim 4 , wherein the performance measurements are generated for the collection period.

7. The method of claim 1, wherein the at least one statistical hypothesis testing function comprises a Bayesian sequential testing function.

8. The method of claim 1, wherein the at least one statistical hypothesis testing function comprises a chi-square test function.

9. A non-transitory computer-readable storage medium, the at least one non-transitory computer-readable storage medium being configured to store instructions that, when executed by at least one processor included in a computing device, cause the computing device to implement scalable variant testing by performing steps comprising: receiving, for a variant test associated with a control feature, a variant test input indicating user information and at least one variant feature; identifying, based on the user information, a control object corresponding to a control feature associated with the user information; generating at least one variant object based on the at least one variant feature; collecting transformation data of the control object and transformation data of the at least one variant object; storing the transformation data of the control object and the transformation data of the at least one variant object in a fact database according to a collection date; retrieving from the fact database a subset of transformation data associated with the control object and a subset of the transformation data associated with the at least one variant object; calculating at least one conversion metric for the control object using the subset of conversion data associated with the control object; calculating at least one transformation metric for the at least one variant object using the subset of transformation data associated with the control object; generating a performance measure for the variant test by applying at least one statistical hypothesis testing function to the at least one transformation metric for the control subject and the at least one transformation metric for the at least one variant subject; as well as The performance measurements are provided at a display.

10. The non-transitory computer-readable storage medium of claim 9, wherein: generating the conversion data for the control object based on the number of unique users accessing the control object, and For each corresponding user associated with the number of unique users accessing the control object, one of the following: a first value indicating that the corresponding user downloaded an application associated with the control object when the corresponding user first accessed the control object, and a second value indicating that the corresponding user did not download the application associated with the control object when the corresponding user first accessed the control object.

11. The non-transitory computer-readable storage medium of claim 9, wherein: generating the conversion data for the at least one variant object based on a number of unique users accessing the at least one variant object, and For each corresponding user associated with the number of unique users accessing the at least one variant object, one of: a first value indicating that the corresponding user downloaded an application associated with the at least one variant object when the corresponding user first accessed the at least one variant object, and a second value indicating that the corresponding user did not download the application associated with the at least one variant object when the corresponding user first accessed the at least one variant object.

12. The non-transitory computer-readable storage medium of claim 9, wherein the subset of transformation data associated with the control object and the subset of transformation data associated with the at least one variant object correspond to a collection cycle.

13. The non-transitory computer-readable storage medium of claim 12, wherein the collection period comprises one day.

14. The non-transitory computer-readable storage medium of claim 12, wherein the performance measurement is generated for the collection cycle.

15. The non-transitory computer-readable storage medium of claim 9, wherein the at least one statistical hypothesis testing function comprises a Bayesian sequential testing function.

16. The non-transitory computer-readable storage medium of claim 9, wherein the at least one statistical hypothesis testing function comprises a chi-square test function.

17. A computing device configured to implement scalable variant testing, the computing device comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the computing device to perform steps comprising: receiving, for a variant test associated with a control feature, a variant test input indicating user information and at least one variant feature; identifying, based on the user information, a control object corresponding to a control feature associated with the user information; generating at least one variant object based on the at least one variant feature; collecting transformation data of the control object and transformation data of the at least one variant object; storing the transformation data of the control object and the transformation data of the at least one variant object in a fact database according to a collection date; retrieving from the fact database a subset of transformation data associated with the control object and a subset of the transformation data associated with the at least one variant object; calculating at least one conversion metric for the control object using the subset of conversion data associated with the control object; calculating at least one transformation metric for the at least one variant object using the subset of transformation data associated with the control object; generating a performance measure for the variant test by applying at least one statistical hypothesis testing function to the at least one transformation metric for the control subject and the at least one transformation metric for the at least one variant subject; as well as The performance measurements are provided at a display.

18. The computing device of claim 17, wherein: generating the conversion data for the control object based on the number of unique users accessing the control object, and For each corresponding user associated with the number of unique users accessing the control object, one of the following: a first value indicating that the corresponding user downloaded an application associated with the control object when the corresponding user first accessed the control object, and a second value indicating that the corresponding user did not download the application associated with the control object when the corresponding user first accessed the control object.

19. The computing device of claim 17, wherein: generating the conversion data for the at least one variant object based on a number of unique users accessing the at least one variant object, and For each corresponding user associated with the number of unique users accessing the at least one variant object, one of: a first value indicating that the corresponding user downloaded an application associated with the at least one variant object when the corresponding user first accessed the at least one variant object, and a second value indicating that the corresponding user did not download the application associated with the at least one variant object when the corresponding user first accessed the at least one variant object.

20. The computing device of claim 17, wherein the subset of transformation data associated with the control object and the subset of transformation data associated with the at least one variant object correspond to a collection cycle.

21. The computing device of claim 20, wherein the collection period comprises a day.

22. The computing device of claim 20, wherein the performance measurement is generated for the collection cycle.

23. The computing device of claim 17, wherein the at least one statistical hypothesis testing function comprises a Bayesian sequential testing function.

24. The computing device of claim 17, wherein the at least one statistical hypothesis testing function comprises a chi-square test function.

25. A computing device configured to implement scalable variant testing, the computing device comprising: means for receiving a variant test input indicating user information and at least one variant feature for a variant test associated with a control feature; means for identifying, based on the user information, a control object corresponding to a control feature associated with the user information; means for generating at least one variant object based on the at least one variant feature; means for collecting transformation data of the control object and transformation data of the at least one variant object; means for storing the transformation data of the control object and the transformation data of the at least one variant object in a fact database according to a collection date; means for retrieving from said fact database a subset of transformation data associated with said control object and a subset of said transformation data associated with said at least one variant object; means for calculating at least one conversion metric for the control object using the subset of conversion data associated with the control object; means for calculating at least one transformation metric for said at least one variant object using said subset of transformation data associated with said control object; means for generating a performance measure for the variant test by applying at least one statistical hypothesis testing function to the at least one transformation metric for the control subject and the at least one transformation metric for the at least one variant subject; and Means for providing said performance measurement at a display.

26. The computing device of claim 25, wherein: generating the conversion data for the control object based on the number of unique users accessing the control object, and For each corresponding user associated with the number of unique users accessing the control object, one of the following: a first value indicating that the corresponding user downloaded an application associated with the control object when the corresponding user first accessed the control object, and a second value indicating that the corresponding user did not download the application associated with the control object when the corresponding user first accessed the control object.

27. The computing device of claim 25, wherein: generating the conversion data for the at least one variant object based on a number of unique users accessing the at least one variant object, and For each corresponding user associated with the number of unique users accessing the at least one variant object, one of: a first value indicating that the corresponding user downloaded an application associated with the at least one variant object when the corresponding user first accessed the at least one variant object, and a second value indicating that the corresponding user did not download the application associated with the at least one variant object when the corresponding user first accessed the at least one variant object.

28. The computing device of claim 25, wherein the subset of transformation data associated with the control object and the subset of transformation data associated with the at least one variant object correspond to a collection cycle.

29. The computing device of claim 28, wherein the collection period comprises a day.

30. The computing device of claim 28, wherein the performance measurement is generated for the collection cycle.

31. The computing device of claim 25, wherein the at least one statistical hypothesis testing function comprises a Bayesian sequential testing function.

32. The computing device of claim 25, wherein the at least one statistical hypothesis testing function comprises a chi-square test function.

Images