BALANCED CONTROL TREATMENT EXPERIMENTS FOR SOFTWARE TESTS

Abstract

A computing system can determine that a software application installed on a computing device in a control group associated with a control treatment experiment for the software application is a control version of the software application. Based at least in part on the fact that the computing device is in the control group and that the software application installed on the computing device is the control version of the software application, the computing system can determine a software update to be sent to the computing device to update the software application. The computing system can send the software update to the computing device to trigger the computing device to reinstall the control version of the software application from a storage device on the computing device in response to receiving the software update.

Description

BACKGROUND

[0001] A software developer may conduct control treatment experiments (i.e., A / B testing) to evaluate whether performance issues encountered when downloading, installing, and / or running a new version of a software application were caused by the new version of the software application. The software developer may collect performance data for the new version of the software application and performance data for an older version of the software application and use the collected performance data to identify performance issues caused by the new version of the software application. SUMMARY

[0002] In general, the techniques of this disclosure are directed to conducting control treatment experiments for a software application in such a way as to improve the performance of such experiments. A computing system may deploy a treatment version of a software application on computing devices in a treatment group, and the performance of the treatment version of the software application may be compared to the performance of a control version of the software application installed on computing devices in the control group.

[0003] The computing system may collect performance data regarding the download and installation of the treatment version of the software application on computing devices. To generate corresponding performance data from the control group computing devices, the control group computing device may send the treatment version of the software application to the control group computing devices that have installed the control version of the software application. The control group computing devices may reinstall the control version of the software application in response to receiving the treatment version of the software application.

[0004] Because both the computing devices in the control group and the computing devices in the treatment group can download the treatment version of the software application, the techniques of this disclosure can reduce any tendency to compare the performance data related to the computing devices in the control group and the performance data related to the computing devices in the treatment group, which may be caused by the computing devices in the control group and the computing devices in the treatment group downloading different data that have very different characteristics (e.g., different download sizes).Furthermore, by causing the computing devices in the control group to reinstall the control version of the software application, the techniques of this disclosure can trigger the computing devices to perform an installation action, allowing the performance data related to the installation of the treatment version of the software application on computing devices in the treatment group to be compared with corresponding performance data related to the reinstallation of the control version of the software application on computing devices in the control group. In this way, the techniques of the present disclosure enhance the technical field of software testing.

[0005] Because computing devices in the control group download the treatment version of the software application when the treatment version of the software application becomes the new release version of the software application, the computing devices in the control group may not need to re-download the treatment version of the software application to upgrade to the new release version of the software application. In this way, the techniques of this disclosure may also reduce network usage by the computing devices in the control group, potentially providing further technical improvements.

[0006] In some aspects, the techniques described herein relate to a method including: determining, by one or more processors of a computing system, that a software application installed on a computing device in a control group associated with a control treatment experiment for the software application is a control version of the software application; determining, by the one or more processors and based at least in part on the computing device being in the control group and the software application installed on the computing device being the control version of the software application, a software update to update the software application to a treatment version of the software application to be sent to the computing device;and sending, by the one or more processors, to the computing device the software update to trigger the computing device to reinstall the control version of the software application on the computing device from a storage device of the computing device in response to receiving the software update;

[0007] In some aspects, the techniques described herein relate to a computing system including: memory storing instructions; and one or more processors executing the instructions to: determine that a software application installed on a computing device in a control group associated with a control treatment experiment for the software application is a control version of the software application; determine a software update to update the software application to a treatment version of the software application to be sent to the computing device based at least in part on the computing device being in the control group and the software application installed on the computing device being the control version of the software application;and sending the software update to the computing device to trigger the computing device to reinstall the control version of the software application on the computing device from a storage device of the computing device in response to receiving the software update;

[0008] In some aspects, the techniques described herein relate to a non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors of a computing system to: determine that a software application installed on a computing device in a control group associated with a control treatment experiment for the software application is a control version of the software application; determine a software update to update the software application to a treatment version of the software application to be sent to the computing device based at least in part on the computing device being in the control group and the software application installed on the computing device being the control version of the software application;and sending the software update to the computing device to trigger the computing device to reinstall the control version of the software application on the computing device from a storage device of the computing device in response to receiving the software update;

[0009] In some aspects, the techniques described herein relate to an apparatus including: means for determining that a software application installed on a computing device in a control group associated with a control treatment experiment for the software application is a control version of the software application; means for determining a software update to update the software application to a treatment version of the software application to be sent to the computing device based at least in part on the computing device being in the control group and the software application installed on the computing device being the control version of the software application;and means for sending the software update to the computing device to trigger the computing device to reinstall the control version of the software application on the computing device from a storage device of the computing device in response to receiving the software update;

[0010] In some aspects, the techniques described herein relate to a method including: receiving, by one or more processors of a computing device and from a computing system, a software update for updating a software application, wherein the computing device is in a control group associated with a control treatment experiment for the software application, and wherein a control version of the software application is installed on the computing device; determining, by the one or more processors, whether the software update is for updating the software application to a treatment version of the software application;and in response to determining that the software update is to update the software application to the treatment version of the software application, reinstalling, by the one or more processors, the control version of the software application on the computing device from a storage device of the computing device;

[0011] In some aspects, the techniques described herein relate to a computing system including: a memory storing instructions; a storage device; and one or more processors executing the instructions to: receive a software update from a computing system to update a software application, wherein the computing device is in a control group associated with a control treatment experiment for the software application, and wherein a control version of the software application is installed on the computing device; determine whether the software update is to update the software application to a treatment version of the software application;and in response to determining that the software update is for updating the software application to the treatment version of the software application, reinstalling the treatment version of the software application on the computing device from the storage device of the computing device;

[0012] In some aspects, the techniques described herein relate to a non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors of a computing device to receive, from a computing system, a software update for updating a software application, wherein the computing device is in a control group associated with a control treatment experiment for the software application, and wherein a control version of the software application is installed on the computing device; determining whether the software update is for updating the software application to a treatment version of the software application;and in response to determining that the software update is for updating the software application to the treatment version of the software application, reinstalling the control version of the software application on the computing device from a storage device of the computing device;

[0013] In some aspects, the techniques described herein relate to an apparatus including: means for receiving a software update to update a software application from a computing system, wherein the computing device is in a control group associated with a control treatment experiment for the software application, and wherein a control version of the software application is installed on the computing device; means for determining whether the software update is to update the software application to a treatment version of the software application; and means for reinstalling the control version of the software application on the computing device from a storage device of the computing device in response to determining that the software update is to update the software application to the treatment version of the software application.

[0014] The details of one or more examples are set forth in the accompanying drawings and the description below. Further features, objects, and advantages of the disclosure will become apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a conceptual diagram illustrating an example software testing environment, according to one or more aspects of the present disclosure. Fig. 2 is a block diagram illustrating further details of an example computing device according to one or more aspects of the present disclosure. Fig. 3 is a block diagram illustrating an example computing system according to one or more aspects of the present disclosure. Fig. 4A-4B are block diagrams illustrating software testing according to aspects of this disclosure. Fig. 5 is a flowchart illustrating example operations performed by an example computing system configured to perform software testing in accordance with one or more aspects of the present disclosure. Fig. 6 is a flowchart illustrating example operations performed by an example computing device located in a control group for a control treatment experiment to test a software application, according to one or more aspects of the present disclosure. DETAILED DESCRIPTION

[0015] Fig. 1 is a conceptual diagram illustrating an example software testing environment, according to one or more aspects of the present disclosure. In the example of Fig. 1, the environment 100 may include a computing system 102 that communicates with the computing devices 110-1A through 110-1N and the computing devices 110-2A through 110-2N (collectively, “computing devices 110”) via a network 130.

[0016] Computing system 102 may represent any suitable computing system, such as one or more desktop computers, laptops, mainframes, servers, cloud computing systems, etc., capable of both sending and receiving information to and from a network, such as network 130. In some examples, computing system 102 may represent cloud computing systems that provide access to their respective services via a cloud.

[0017] Computing system 102 includes a test module 104, an application distribution module 106, and a test data store 108. Test module 104 and application distribution module 106 may perform the operations described herein using software, hardware, firmware, or a mixture of hardware, software, and firmware residing and / or executing within computing system 102. Computing system 102 may execute test module 104 and application distribution module 106 with multiple processors or multiple devices, as virtual machines executing on underlying hardware, as one or more services of an operating system or computing platform, and / or as one or more executable programs at an application layer of a computing platform of computing system 102.

[0018] The testing module 104 may perform functions associated with conducting control treatment experiments (i.e., A / B testing) for software applications installed on computing devices 110. In some examples, the testing module 104 may be included in the operating system of the computing system 102.

[0019] The application distribution module 106 may perform functions associated with distributing software applications and / or software updates to computing devices 110. For example, the application distribution module 106 may send software updates to computing devices 110 to update software applications (e.g., software application 122). Furthermore, the application distribution module 106 may handle requests from computing devices 110 to automatically update and / or manually update software applications by sending software application updates to the computing devices 110. In some examples, the application distribution module 106 may be included in the operating system of the computing system 102.

[0020] Test data storage 108 may be any suitable data storage, such as a database, directory, and the like, for storing data associated with the performance of control treatment experiments for software applications. For example, test data storage 108 may store information about the versions of software application 122 installed on each computing device 110, the test status of the control treatment experiment, information about the treatment version of software application 122, information about the control version of software application 122, and the like.

[0021] Network 130 represents any public or private communications network, such as a cellular network, Wi-Fi, and / or other types of networks for transferring data between computing systems, servers, and computing devices. Network 130 may include one or more network hubs, network switches, network routers, or other network equipment operatively coupled to each other, thereby providing for the exchange of information between computing system 102 and computing devices 110. Computing devices 110 and computing system 102 may transmit and receive data over network 130 using any suitable communication techniques. Each computing system 102 and computing device 110 is operatively coupled to network 130 using appropriate network connections, such as Ethernet, Wi-Fi, or other types of wired and / or wireless network connections.

[0022] The computing devices 110 each represent an individual mobile or non-mobile computing device. Examples of each of the computing devices 110 include a mobile phone, a tablet computer, a laptop computer, a desktop computer, a server, a mainframe, a set-top box, a television, a wearable device (e.g., a computerized watch, computerized glasses, computerized headphones, computerized gloves, etc.), a home automation device or system (e.g., a smart thermostat or smart home assistant device), a personal digital assistant (PDA), a gaming system, a media player, an e-book reader, a mobile television platform, an automotive navigation or infotainment system, or any other type of mobile, non-mobile, portable, and non-wearable computing device.

[0023] Each of the computing devices 110 includes an application management module 116 and a software application 112. The application management module 116 and the software application 112 may perform the operations described herein using software, hardware, firmware, or a mixture of hardware, software, and firmware residing on and / or executing at the computing device 110-1A. Each of the computing devices 110 may execute the application management module 116 and the software application 112 with multiple processors or multiple devices, as virtual machines executing on the underlying hardware, as one or more services of an operating system or computing platform, and / or as one or more executable programs on an application layer of a computing platform of each of the computing devices 110.

[0024] The software application 122 may be any software application that can be executed by the computing devices 110 to perform one or more functions. In some examples, the software application 122 may be a native application provided by the operating system or by the manufacturer of the computing device 110. In some examples, the software application 122 may be an application provided by a third-party developer.

[0025] The application management module 116 may perform functions associated with managing software applications, including software application 122, installed on computing devices 110. For example, the application management module 116 may download (e.g., from the computing system 102) and install software applications, reinstall applications already installed on computing devices 110, perform manual and / or automatic updates of software applications, and the like. In some examples, the application management module 116 may be included in the operating system of computing devices 110 or may be part of an app store application on computing devices 110.

[0026] In the example of Fig. 1, a developer may use environment 100 to conduct control treatment experiments (i.e., A / B testing) to test versions of a software application (e.g., software application 122) before such versions of the software application are potentially released. A developer may conduct such a control treatment experiment to evaluate whether performance issues encountered when downloading, installing, and / or running a version of the software application were caused by the version of the software application.

[0027] A developer of a software application may periodically release new versions of a software application to provide new features, fix bugs in previous versions of the software application, and the like. However, in some instances, the new version of the software application may cause problems that adversely affect the performance of computing devices that have the new version of the software application installed. For example, computing devices may fail to successfully download and / or install the new version of the software application, the new version of the software application may crash during use, the new version of the software application may introduce new bugs, and the like.

[0028] The developer of the software application may conduct control treatment experiments with a version of the software application to evaluate whether performance issues encountered when downloading, installing, and / or running that version of the software application were caused by the version of the software application. To conduct such experiments, a developer may conduct a controlled release of the new version of the software application to roll out the new version of the software application to a limited number of computing devices. That is, the developer may release the new version of the software application to fewer than all computing devices on which the software application is installed.

[0029] By performing a controlled release of the new version of the software application, the developer can divide the computing devices on which the software application is installed into two groups of equal or very similar size: a treatment group containing computing devices on which the new version of the software application is installed, and a control group containing computing devices on which an older version of the software application is installed. In the context of control treatment experiments, the new version of the software application under test is referred to as the treatment version of the software application. Furthermore, the control group may include computing devices on which the same older version of the software application is installed, and this older version of the software application is referred to here as the control version of the software application in the context of control treatment experiments.

[0030] The developer of the software application may collect performance data associated with the software application from both the computing devices in the treatment group on which the treatment version of the software application is installed and the computing devices in the control group on which the control version of the software application is installed. Such performance data may include the download failure rate, the installation failure rate, the application crash rate, the application launch latency, power consumption, memory utilization, the frame rate metric, or other suitable performance data and / or metrics.The developer can compare the performance data collected from the computing devices in the treatment group with the performance data collected from the computing devices in the control group to compare the performance issues associated with the treatment version of the computing device with the performance issues associated with the control version of the software application. This allows the developer to determine the performance issues associated with the new version of the software application and evaluate whether such performance issues were caused by the new version of the software application.

[0031] Performance issues associated with the release of a new version of a software application can occur not only during startup or execution of the software application. Instead, the release of a new version of a software application can sometimes cause performance issues during downloading and installing the software application. Thus, a developer can also collect performance data associated with the download and installation of the treatment version of the software application by computing devices in the treatment group.

[0032] To compare the performance data associated with the downloading and installation of the treatment version of the software application on computing devices in the treatment group with the corresponding data generated by computing devices in the treatment group, the developer can generate a placebo version of the software application, and computing devices in the control group can download and install the placebo version of the software application, which can then become the control version of the software application. The developer is therefore able to collect performance data associated with the downloading and installation of the placebo version of the software application by the computing devices in the control group to compare it with the performance data associated with the downloading and installation of the treatment version of the software application by the computing devices in the treatment group.

[0033] However, using such placebo versions of the software application to generate performance data for the control group to compare with the performance data for the treatment group can cause certain technical problems. For example, computing devices may not know that the placebo version of the software application is an upgrade and thus be unable to install the placebo version of the software application. This may require the developer to create designs of placebo versions of the software application with very specific version codes so that the computing devices can recognize that the placebo version of the software application is an upgrade.

[0034] Furthermore, failure to upgrade to the placebo version of the software application may result in the computing devices in the control group running different versions of the software application, creating an imbalance between the number of computing devices running the treatment versions of the software application and the number of computing devices running the placebo version of the software application in the control group. Such an imbalance, also referred to as arm imbalance, may potentially distort the comparison of performance data between the treatment and control groups.

[0035] Furthermore, a software update to update the software application to a placebo version of the software application may differ in size from a software update to update the software application to a treatment version of the software application. The different sizes of the software updates may result in variations in the download and installation performance of the placebo version of the software application versus the treatment version of the software application, potentially distorting or influencing comparisons of the performance data associated with downloading and installing the placebo version of the software application and the performance data associated with downloading and installing the treatment version of the software application.

[0036] According to aspects of the present disclosure, a computing system may use a version of the software application as a placebo to compare the performance data associated with the control group and the performance data associated with the treatment group when conducting control treatment experiments for a software application. That is, in addition to deploying a treatment version of a software application to computing devices in the treatment group, the computing system may also send a version of the software application to computing devices in the control group to collect and compare performance data related to the treatment version of the software application deployed on the treatment group and the control version of the software application deployed on the control group.

[0037] In particular, a computing system may determine, for a computing device in the control group, the version of the software application installed on the computing device. If the computing system determines that the version of the software application installed on the computing device is the control version of the software application, the computing system may send the computing device's treatment version to the computing device to trigger the computing device to reinstall the control version on the computing device. In response to receiving the treatment version of the software application, the computing device may reinstall the control version of the software application from a storage device of the computing device.

[0038] Because both the computing devices in the control group and the computing devices in the treatment group can download the treatment version of the software application, the techniques of this disclosure can reduce any tendency to compare the performance data related to the computing devices in the control group and the performance data related to the computing devices in the treatment group, which may be caused by the computing devices in the control group and the computing devices in the treatment group downloading different data that have very different characteristics (e.g., different download sizes).Furthermore, by causing the computing devices in the control group to reinstall the control version of the software application, the techniques of this disclosure can trigger the computing devices to perform an installation action, allowing the performance data related to the installation of the treatment version of the software application on computing devices in the treatment group to be compared with corresponding performance data related to the reinstallation of the control version of the software application on computing devices in the control group. In this way, the techniques of the present disclosure enhance the technical field of software testing.

[0039] Similarly, if the computing system determines that the version of the software application installed on the computing device is not the control version of the software application, the computing system may send the control version of the computing device to the computing device to trigger the computing device to install the control version on the computing device. In response to receiving the control version of the software application, the computing device may install the control version of the software application by sending and installing the control version of the software application to computing devices in the control group that do not have the control version of the software application installed.In this way, the techniques of this disclosure can increase the number of computing devices in the control group on which the control version of the software application is installed, thereby improving the security of the computing devices in the control group by increasing the number of computing devices upgraded from older versions of the software application to the control version of the software application. Furthermore, increasing the number of computing devices in the control group on which the control version of the software application is installed can reduce the difference between the number of computing devices in the treatment group on which the treatment version of the software application is installed and the number of computing devices in the control group on which the control version of the software application is installed, thereby improving the technical field of software testing.

[0040] In the example of Fig. 1, to perform such testing of the software application 122, the developer may divide the computing devices 110 with which such testing is to be performed into two groups: the control group 140 and the treatment group 142. The computing devices 110-1A through 110-1N ("computing device 110-1") are included in the control group 140, while the computing devices 110-2A through 110-2M ("computing device 110-2") are included in the treatment group 142.

[0041] Computing devices 110-2 in treatment group 142 are computing devices that are intended to receive and install new versions of software application 122 for testing, while computing devices 110-1 in control group 140 can execute a control version of software application 122. The developer can compare performance data associated with the new versions of software application 122 collected from computing devices 110-2 in treatment group 142 with performance data associated with the control version of software application 122 collected from computing devices 110-1 in control group 140 to determine performance issues associated with the new versions of software application 122 and to evaluate whether such performance issues were caused by the new versions of software application 122.

[0042] The developer may arbitrarily select the computing devices 110-1 to be included in the control group 140 from the computing devices 110, and the computing devices 110-2 to be included in the treatment group 142 from the computing devices 110. In some examples, the developer may select the computing devices 110-1 to be included in the control group 140 and the computing devices 110-2 to be included in the treatment group 142, such that the control group 140 and the treatment group 142 include the same number of computing devices.

[0043] In some examples, computing system 102 may use one or more heuristics, such as implemented by one or more neural networks, to select computing devices 110-1 to be included in control group 140 from computing devices 110 and computing devices 110-2 to be included in treatment group 142 from computing devices 110. The one or more neural networks may be implemented by computing system 102 or implemented by a remote computing system in communication with computing system 102.

[0044] In general, one or more neural networks may include multiple interconnected nodes, and each node may apply one or more functions to a set of input values ​​corresponding to one or more features and provide one or more corresponding output values.In the context of selecting computing devices 110 to be included in the control group 140 and / or the treatment group 142, the one or more features may include usage data of the software application 122 on computing devices 110, usage data of the computing devices 110, performance data collected from computing devices 110 during previous control treatment experiments, and / or any other data related to computing devices 110. The one or more corresponding output values ​​of one or more neural networks may be an indication, for each of the computing devices 110, of whether the computing device is included in the control group 140, whether the computing device is included in the treatment group 142, or whether the computing device is not to be included in either the control group 140 or the treatment group 142.

[0045] Computing system 102 may determine the control version of software application 122 for the control treatment experiment in any manner. In general, the control version of software application 122 and the treatment version of software application 122 may be the same as or a newer version than the versions of software application 122 installed on computing devices 110. In some examples, computing system 102 may determine the control version of software application 122 as a version of software application 122 that is fully released to computing devices (e.g., the most commonly installed version of software application 122 on computing devices 110) or as the most recent version of software application 122 prior to the new version of software application 122 currently being tested.In some examples, the control version of the software application 122 may be a developer-created version of the software application 122 that is most similar in size to, or includes the same modules as, the new version of the software application 122.

[0046] To test a new version of the software application 122, the developer may deploy the new version of the software application 122 on each of the computing devices 110-2 in the treatment group 142. In control treatment experiments, the new version of the software application 122 being tested is referred to as the treatment version of the software application 122. The developer may use the computing system 102 to send the treatment version of the software application 122 to each of the computing devices 110-2 in the treatment group 142, and each of the computing devices 110-2 in the treatment group 142 may receive and install the treatment version of the software application 122, such as by using the application management module 116 to receive and install a software upgrade to upgrade the software application 122 installed on computing devices 110-2 to the treatment version of the software application 122.

[0047] Computing system 102 may send the treatment version of software application 122 to computing devices 110 in any form. For example, computing system 102 may send a software update to computing devices 110-2 in treatment group 142 to update software application 122 to the treatment version of software application 122. The software update may include a package, an archive file, or other technique for bundling code and resources, and may be compressed or uncompressed.

[0048] In some examples, a software update may include code and data to install the treatment version of the software application 122 instead of the currently installed version of the software application 122. That is, in response to receiving a software update, the application management module 116 may replace the currently installed version of the software application 122 with the treatment version of the software application 122.

[0049] In some examples, a software update may include code and data to patch the currently installed version of the software application 122 to upgrade the software application 122 to the treatment version of the software application 122. That is, in response to receiving the software update, the application management module 116 may patch the currently installed version of the software application 122 to update the currently installed version of the software application 122 to the treatment version of the software application 122.

[0050] In some examples, a software update may include modular updates to the software application 122. Such modular updates may update certain features and functionality of the software application 122, referred to herein as modules, to the treatment version of those features and functionality. A software update to update one or more modules of the software application 122 may include code and data to install the treatment version of the one or more modules in a currently installed version of the software application 122 and / or may include code and data to patch the one or more modules in a currently installed version of the software application 122 to update those one or more modules in the software application 122 to the treatment version of the one or more modules.

[0051] As part of testing a treatment version of the software application 122, the computing system 102 may also send data to the computing devices 110-1 in the control group 140. Sending data to the computing devices 110-1 in the control group 140 may act as a placebo corresponding to the treatment version of the software application 122 and allow the developer to properly evaluate the performance of the computing devices in the treatment group 142 downloading and installing the treatment version of the software application versus the computing devices in the control group 140 downloading and installing data.

[0052] According to aspects of the disclosure, during a control treatment experiment for the software application 122, the testing module 104 of the computing system 102 may determine the version of the software application 122 currently installed on the computing device for a computing device from the computing devices 110-1 in the control group 140. The testing module 104 may determine which version of the software application 122 to send to the computing device based on the currently installed version of the software application 122 on the computing device. The application distribution module 106 may therefore send the determined version of the software application 122 to the computing device.

[0053] Sending a version of the software application 122 to the computing devices 110-1 in the control group 140 may allow a developer of the software application 122 to compare the performance of the computing devices 110-2 in the treatment group 142 in downloading and installing the treatment version of the software application 122 with the performance of the computing device 110-1 in the control group 140 in downloading and installing versions of the software application 122. That is, if the computing system 102 receives a request to update the software application 122 from a computing device in the control group 140 during the control treatment experiment, such asan automatic update request or a manual update request, the computing system 102 may responsively send a version of the software application 122 to the requesting computing device, and the developer may collect performance data regarding the download of the software application 122.

[0054] In some examples, computing system 102 may send the treatment version of software application 122 to computing devices 110-1 in control group 140 that already have the control version of software application 122 installed to trigger those computing devices to reinstall the control version of software application 122 from a storage device of the computing devices. That is, if computing system 102 determines that a computing device in control group 140 already has the control version of software application 122 installed, computing system 102 may determine that the control version of software application 122 should remain installed on the computing device.

[0055] Thus, in response to determining that the control version of the software application 122 is installed on a computing device in the control group 140, the computing system 102 may send the treatment version of the software application 122 to the computing device, causing the computing device to reinstall the control version of the software application 122 from a storage device of the computing devices. Triggering the reinstallation of the control version of the software application 122 may therefore allow the developer of the software application 122 to collect performance data regarding the installation of the control version of the software application 122 on computing devices 110-1 in the control group 140 while the control version of the software application 122 may remain installed on the computing device.

[0056] Similarly, computing system 102 may send the control version of software application 122 to computing devices 110-1 in control group 140 that do not have the control version of software application 122 installed. Sending the control version of software application 122 to computing devices 110-1 in control group 140 that do not have the control version of software application 122 installed may trigger those computing devices to upgrade the installed version of software application 122 to the control version of software application 122.Thus, in response to determining that the version of the software application 122 installed on a computing device in the control group 140 is not the control version of the software application 122, the computing system 102 may send the control version of the software application 122 to the computing device, causing the computing device to install the control version of the software application 122 from a storage device of the computing devices.

[0057] In this way, the computing system 102 may be able to increase the number of computing devices 110-1 in the control group 140 on which the control version of the software application 122 is installed. Increasing the number of computing devices 110-1 in the control group 140 on which the control version of the software application 122 is installed may reduce an imbalance between the number of computing devices 110-1 in the control group 140 on which the control version of the software application 122 is installed and the number of computing devices 110-2 in the treatment group 142 on which the treatment version of the software application 122 is installed.

[0058] Computing system 102 may determine the version of software application 122 installed on computing devices 110-1 in control group 140 in any suitable manner. For example, test data storage 108 may store information about the version of software application 122 installed on each of computing devices 110, and computing system 102 may use test data storage 108 to look up the version of software application 122 installed on computing devices 110-1 in control group 140, as indicated by test data storage 108.

[0059] Computing system 102 may send the particular version of software application 122 to computing devices 110 in any form. As described above, computing system 102 may send the particular version of software application 122 in the form of a software update to update software application 122 to the particular version of software application 122. The software update may include a package, an archive file, or other technique for bundling code and resources, and may be compressed or uncompressed.

[0060] In some examples, a software update may include code and data to install the particular version of the software application 122 instead of the currently installed version of the software application 122. That is, in response to receiving a software update, the application management module 116 may replace the currently installed version of the software application 122 with the particular version of the software application 122.

[0061] In some examples, a software update may include code and data to patch the currently installed version of the software application 122 to upgrade the software application 122 to the particular version of the software application 122. That is, in response to receiving the software update, the application management module 116 may patch the currently installed version of the software application 122 to upgrade the currently installed version of the software application 122 to the particular version of the software application 122.

[0062] In some examples, a software update may include modular updates to the software application 122. Such modular updates may update certain features and functionality of the software application 122, referred to herein as modules, to the particular version of those features and functionality. A software update to update one or more modules of the software application 122 may include code and data to install the particular version of the one or more modules in a currently installed version of the software application 122 and / or may include code and data to patch the one or more modules in a currently installed version of the software application 122 to update those one or more modules in the software application 122 to the particular version of the one or more modules.

[0063] In the example of Fig. 1, computing system 102 may receive a request from computing device 110-1A in control group 140 to update the software update of software application 122 installed on computing device 110-1A. In response, computing system 102 may determine the version of software application 122 installed on computing device 110-1A. If computing system 102 determines that the version of software application 122 installed on computing device 110-1A is the control version of software application 122, computing system 102 may send a software update to computing device 110-1A to update software application 122 to the treatment version of software application 122 and trigger computing device 110-1A to reinstall the control version of software application 122 from a storage device of computing device 110-1A.That is, the computing device 110-1A may store data for reinstalling the control version of the software application 122, such as an installation package, an application package, etc., on the hard disk, and receipt of the treatment version of the software application 122 may cause the computing device 110-1A to reinstall the control version of the software application 122 from a storage device of the computing device 110-1A.

[0064] Similarly, if computing system 102 determines that the version of software application 122 installed on computing device 110-1A is not the control version of software application 122, computing system 102 may responsively send a software update to computing device 110-1A to update software application 122 to the control version of software application 122. Computing device 110-1A, in response to receiving the software update to update software application 122 to the control version of software application 122, may use the received software update to update software application 122 installed on computing device 110-1A to the control version of software application 122.

[0065] In response to receiving the software update, computing device 110-1A may determine whether to reinstall the currently installed version of software application 122 from the storage device of computing device 110-1A or upgrade to the received version of software application 122, based at least in part on determining whether the computing device is in a test state, also referred to as a canary state. Computing devices 110-1A may be in a test state when a control treatment experiment is being conducted to test a treatment version of software application 122 and may not be in the test state when no control treatment experiment is being conducted to test a treatment version of software application 122.If the computing device 110-1A determines that the computing device is not in the test state, the computing device may install the version of the software application 122 from the computing system 102.

[0066] Computing device 110-1A may determine whether the computing device is in a test state using any suitable technique. In some examples, computing device 110-1A may receive an indication from computing system 102 regarding whether computing device 110-1A is in a test state along the software update. In some examples, computing device 110-1A may store an indication of whether computing device 110-1A is in the test state, which computing device 110-1A may update each time computing device 110-1A enters or exits the test state. In some examples, computing device 110-1A may query computing system 102 regarding whether computing device 110-1A is in the test state and may receive an indication from computing system 102 regarding whether computing device 110-1A is in the test state.

[0067] If the computing device 110-1A determines that the computing device is in the test state, the computing device 110-1A may determine whether to reinstall the currently installed version of the software application 122 from the storage device of the computing device 110-1A or upgrade to the received version of the software application 122, based at least in part on the version of the software application 122 associated with the software update received from the computing system 102. If the computing device 110-1A determines that the software update received from the computing system 102 is a software update for updating the software application 122 to the treatment version of the software application 122, the computing device 110-1A may determine to reinstall the currently installed version of the software application 122 from the storage device of the computing device 110-1A.In this way, the computing system 102 may trigger the computing device 110-1A to reinstall the currently installed version of the software application 122 from the storage device of the computing device 110-1A by sending a software update to the computing device 110-1A to update the software application 122 to the treatment version of the software application 122.

[0068] In some examples, in addition to reinstalling the currently installed version of the software application 122 from the storage device of the computing device 110-1A, the computing device 110-1A may also use the received update to install a copy of the treatment version of the software application 122 and may store the copy of the treatment version of the software application 122 in a storage device of the computing device 110-1A. Installing and storing a copy of the treatment version of the software application 122 may enable the computing device 110-1A towhen the treatment version of the software application 122 is released as the next official version of the software application 122, upgrade the version of the software application 122 installed on the computing device 110-1A to the treatment version of the software application 122 without having to re-download a software update to upgrade the software application 122 to the treatment version of the software application 122.

[0069] For example, if the treatment version of the software application 122 is released as the next official version of the software application 122, the computing system 102 may send an indication to the computing device 110-1A that the treatment version of the software application 122 is released as the next official version of the software application 122, which triggers the computing device 110-1A to install the copy of the treatment version of the software application 122 stored on the computing device 110-1A to replace the control version of the software application 122 currently installed on the computing device 110-1A.Such an indication sent to the computing device 110-1A need not include a software update to update the software application 122 to the treatment version of the software application 122, since a copy of the treatment version of the software application 122 may already be stored on the computing device 110-1A, thereby reducing the broadband usage of the computing device 110-1A because the computing device 110-1A does not need to re-download a software update to update the software application 122 to the treatment version of the software application 122.

[0070] Similarly, if computing device 110-1A determines that the software update received from computing system 102 is a software update for upgrading software application 122 to the control version of software application 122, computing device 110-1A may also use the software update to upgrade the currently installed version of software application 122 to the control version of software application 122. For example, computing device 110-1A may install the control version of software application 122 to replace the currently installed version of software application 122, update one or more modules of the currently installed version of software application 122 to the treatment versions of the one or more modules, and the like.

[0071] In some examples, regardless of the version of the software application 122 installed on a computing device, the computing system 102 may send a software update to the computing device that includes both code for updating the software application 122 to the treatment version of the software application 122 and code for updating the software application 122 to the control version of the software application 122. A computing device in the control group 140 may use the code for updating the software application 122 to the control version of the software application 122 to update the software application 122 installed on the computing device to the control version of the software application.Similarly, a computing device in the treatment group 142 may use the code for updating the software application 122 to the treatment version of the software application 122 to update the software application 122 installed on the computing device to the treatment version of the software application 122.

[0072] Fig. 2 is a block diagram illustrating further details of an example computing device according to one or more aspects of the present disclosure. The computing device 210 of Fig. 2 is described below as an example of one of the computing devices 110-1 in the control group 140, as in Fig. 1 illustrates.

[0073] In the computing device 210 of Fig. 2 may, for example, be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a server, a mainframe computer, a set-top box, a television, a wearable device, a home automation device or system, a gaming system, a media player, an e-book reader, a mobile television platform, an automotive navigation or infotainment system, or any other type of mobile, non-mobile, wearable, and non-wearable computing device configured to receive and output an indication of notification data. Fig. 2 illustrates only one particular example of the computing device 210, and many other examples of the computing device 210 may be used in other cases and may include a subset of the components included in the example computing device 210, or may include additional components included in Fig. 2 are not shown.

[0074] As in the example from Fig. 2, the computing device 210 includes a UIC 212, one or more processors 240, one or more input components 242, one or more communication units 244, one or more output components 246, and one or more storage components 248. The storage components 248 of the computing device 210 also include an application management module 216, a software application 222, and a software application package 224.

[0075] Communication channels 250 may interconnect each of components 240, 212, 244, 246, 242, and 248 for communication between the components (physical, communicative, and / or operational). In some examples, communication channels 250 may include a system bus, a network connection, a data structure for interprocess communication, or any other method for data communication.

[0076] One or more input components 242 of the computing device 210 can receive inputs. Examples of inputs include tactile, auditory, and visual inputs. The input components 242 of the computing device 210 include, in one example, a presence-sensitive display, a touch-sensitive screen, a mouse, a keyboard, a voice-activated system, a video camera, a microphone, or any other type of device for detecting human or machine inputs.

[0077] One or more output components 246 of the computing device 210 can generate outputs. Examples of outputs include tactile, audible, and visual outputs. The output components 246 of the computing device 210 include, in one example, a presence-sensitive display, a sound card, a video graphics adapter card, a speaker, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a light field display, haptic motors, linearly actuated devices, or any other type of device for generating an output to a human or machine.

[0078] One or more communication units 244 of the computing device 210 can communicate with external devices over one or more wired and / or wireless networks by transmitting and / or receiving network signals on the one or more networks. Examples of one or more communication units 244 include a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device capable of transmitting and / or receiving information. Other examples of one or more communication units 244 can include shortwave radios, cellular data radios, wireless network radios, and Universal Serial Bus (USB) controllers.

[0079] The UIC 212 of the computing device 200 may be hardware that functions as an input and / or output device for the computing device 210. For example, the UIC 212 may include a display component, which may be a screen on which information is displayed by the UIC 212, and a presence-sensitive input component that can detect an object on and / or near the display component.

[0080] One or more processors 240 may implement functionality and / or execute instructions within the computing device 210. For example, one or more processors 240 may receive and execute instructions stored on the computing device 210 from memory components 248 that perform the functionality of modules 216 and 222 and package 224. The instructions executed by one or more processors 240 may cause the computing device 210 to store information in the memory components 248 during program execution. Examples of one or more processors 240 include application processors, display controllers, sensor hubs, and any other hardware configured to function as a processing unit. One or more processors 240 may execute instructions from modules 216 and 222 and package 224 to perform actions or functions.That is, modules 216 and 222 and package 224 may be operated by one or more processors 240 to perform various actions or functions of computing device 210.

[0081] One or more memory components 248 within the computing device 210 may store information for processing during operation of the computing device 210. That is, the computing device 210 may store data accessed by the modules 216 and 222 and the package 224 during execution of the computing device 210. In some examples, the memory component 248 is temporary memory, meaning long-term storage is not a primary purpose of the memory component 248. The memory components 248 on the computing device 210 may be configured as volatile memory for short-term storage of information, and therefore, stored contents are not retained upon power-off. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

[0082] In some examples, the memory components 248 also include one or more computer-readable storage media. The memory components 248 may be configured to store larger amounts of information than volatile memory. The memory components 248 may be further configured for long-term storage of information as non-volatile memory, where information is retained across power-on / off cycles. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable and programmable memory (EEPROM). Memory components 248 may store program instructions and / or information (e.g., data) associated with the modules 216 and 222 and the package 224.The application management module 216 and the software application 222 may be executed in one or more processors 240 to perform similar functions as those of the application management module 116 and the software application 122 of FIG. Fig. 1 to be carried out.

[0083] One or more processors 240 are configured to execute the application management module 216 to perform functions associated with receiving, managing, and otherwise acting on software updates received on the computing device 210. In particular, the application management module 216 may be executed in one or more processors 240 to receive a software update for updating the software application 222 from a computing system (e.g., the computing system 102 of Fig. 1). Since the computing device 210 is in a control group for a control treatment experiment, the application management module 216 may receive either a software update to update the software application 222 to a treatment version of the software application 222 or a software update to update the software application 222 to a control version of the software application 222.

[0084] In response to receiving the software update, the application management module 216 may determine whether to reinstall the currently installed version of the software application 222 or upgrade to the received version of the software application 222. If the application management module 216 determines that the currently installed version of the software application 222 should be reinstalled, the application management module 216 may use the software application package 224 to reinstall the currently installed version of the software application 222 on the computing device 210, where the software application package 224 may be a software installation package for reinstalling the currently installed version of the software application 222.

[0085] In some examples, the application management module 216 may determine, based at least in part on determining whether the computing device 210 is in a test state, also referred to as a canary state, whether to reinstall the currently installed version of the software application 222 from the software application package 224 or to upgrade to the received version of the software application 222. The computing device 210 may be in a test state when a control treatment experiment is being performed on the software application 222 and may not be in the test state when no control treatment experiment is being performed on the software application 222. If the application management module 216 determines that the computing device is not in the test state, the computing device may install the version of the software application 222 that the application management module 216 has received from a computing system (e.g.,the computing system 102 of . Fig. 1) receives.

[0086] The application management module 216 may determine whether the computing device 210 is in a test state using any suitable technique. In some examples, the application management module 216 may receive an indication from a computing system regarding whether the computing device 210 is in a test state along the software update. In some examples, the application management module 216 may store in memory components 248 an indication of whether the computing device 210 is in the test state, which the application management module 216 may update each time the computing device 210 enters or exits the test state. In some examples, the application management module 216 may query a computing system regarding whether the computing device 210 is in the test state and receive a response from the computing system indicating whether the computing device 210 is in the test state.

[0087] If the application management module 216 determines that the computing device is in the test state, the application management module 216 may determine whether to reinstall the currently installed version of the software application 222 from the software application package 224 stored in the storage components 248 or to upgrade the software application 222 using the received software update, based at least in part on the version of the software application 222 associated with the software update received from the computing system.If the application management module 216 determines that the received software update is a software update for updating the software application 222 to the treatment version of the software application 222, the application management module 216 may determine to reinstall the currently installed version of the software application 222 using the software application package 224 stored in the storage components 248. In this way, a computing system may trigger the computing device 210 to reinstall the currently installed version of the software application 222 using the software application package 224 stored in the storage components 248 by sending the computing device 210 a software update for updating the software application 222 to the treatment version of the software application 222.

[0088] In some examples, in addition to reinstalling the currently installed version of the software application 222 using the software application package 224 stored in the storage components 248, the computing device 110-1A may also use the received update to install a copy of the treatment version of the software application 222 and may store the copy of the treatment version of the software application 222 in the storage components 248. Installing and storing a copy of the treatment version of the software application 222 may enable the computing device 210 to upgrade the version of the software application 222 installed on the computing device 210 to the treatment version of the software application 222 at a future time without having to re-download a software update to upgrade the software application 222 to the treatment version of the software application 222.

[0089] Similarly, if the application management module 216 determines that the software update received from a computing system is a software update for upgrading the software application 222 to the control version of the software application 222, the application management module 216 may also use the software update to upgrade the currently installed version of the software application 222 to the control version of the software application 222. For example, the application management module 216 may install the control version of the software application 222 to replace the currently installed version of the software application 222, upgrade one or more modules of the currently installed version of the software application 222 to the treatment versions of the one or more modules, and the like.

[0090] Fig. 3 is a block diagram illustrating an example computing system according to one or more aspects of the present disclosure. The computing system 302 of Fig. 3 is used below as an example for the computing system 102 in Fig. 1 described. Fig. 3 illustrates only one particular example of the computing system 302, and many other examples of the computing system 302 may be used in other instances and may include a subset of the components included in the example computing system 302 or additional components included in Fig. 3. For example, the computing system 302 may include a cluster of servers, and each of the servers forming the cluster of servers comprising the computing system 302 may implement all or some of the components described herein. Fig. 3 described components to perform the techniques disclosed herein.

[0091] As in the example of Fig. 3, the computing system 302 includes one or more processors 340, one or more communication units 342, and one or more storage devices 348. The storage devices 348 include the test module 304, the application distribution module 306, and the test data store 308.

[0092] One or more processors 340 may implement functions and / or execute instructions associated with computing system 310. Examples of one or more processors 340 include application processors, display controllers, auxiliary processors, one or more sensor hubs, and any other hardware configured to function as a processor, control unit, or processing device. Test module 304 and application distribution module 306 may be operated by one or more processors 340 to perform various actions, operations, or functions of computing system 310. For example, one or more processors 340 of computing system 310 may retrieve and execute instructions stored in one or more memory components 348 that cause one or more processors 340 to perform the operations of test module 304 and application distribution module 306.When executed by one or more processors 340, the instructions may cause the computing system 310 to store information in one or more components 348, such as the test data memory 308.

[0093] One or more communication units 342 of the computing system 310 can communicate with external devices over one or more wired and / or wireless networks by transmitting and / or receiving network signals on the one or more networks. Examples of one or more communication units 342 include a network interface card (such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a global positioning satellite (GPS) receiver, or any other type of device capable of transmitting and / or receiving information. Other examples of one or more communication units 342 can include shortwave radios, cellular data radios, wireless network radios, and universal serial bus (USB) controllers.

[0094] Communication channels 350 may interconnect each of components 340, 342, and 348 for communication between the components (physical, communicative, and / or operational). In some examples, communication channels 350 may include a system bus, a network connection, a data structure for interprocess communication, or any other method for data communication.

[0095] One or more storage components 348 within the computing system 310 may store information for processing during operation of the computing system 310 (e.g., the computing system 310 may store data accessed by the testing module 304, the application distribution module 306, and the test data store 308 during execution within the computing system 310). In some examples, one or more storage devices 348 are temporary storage, meaning that a primary purpose of one or more storage devices 348 is not long-term storage. In this example, one or more storage devices 348 may be configured to store information in the short term as volatile memory, such that the stored contents are not retained upon power off.Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

[0096] In some examples, one or more storage devices 348 may also include one or more computer-readable storage media. In some examples, one or more storage devices 348 may include one or more non-transitory computer-readable storage media. One or more storage devices 348 may be configured to store larger amounts of information than in volatile memory. One or more storage devices 348 may further be configured for long-term storage of information as non-volatile memory, where information is retained across power-on / off cycles.Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable and programmable memory (EEPROM). One or more memory devices 348 may store program instructions and / or information (e.g., data) associated with the test module 304, the application distribution module 306, and the test data memory 308. Memory devices 348 may include memory configured to store data or other information associated with the test module 304, the application distribution module 306, and the test data memory 308.

[0097] Test data storage 308 may be any data storage for storing structured and / or unstructured data and may be configured to store all data related to control treatment experiments in software applications. For example, test data storage 308 may store versions of a software application under test, information about the versions of the software applications installed on the computing devices, and the like. Examples of test data storage 308 may include, for example, a database, a schema, a key-value store, or the like.

[0098] One or more processors 340 are configured to execute the application distribution module 306 and the testing module 304 to perform a control treatment experiment for a software application. In particular, one or more processors 340 are configured to execute the application distribution module 306 to distribute software applications to computing devices via one or more communication units 342. In some examples, the application distribution module 306 may generate and send software updates for updating software applications to computing devices via one or more communication units 342. That is, in some examples, when the application distribution module 306 (e.g.,from the testing module 304) receives a request to send a version of a software application to one or more computing devices, the application distribution module 306 can generate a software update to update the software application to the version of the software application and send the software update to the one or more computing devices.

[0099] One or more processors 340 are configured to execute the test module 304 to perform a control treatment experiment for a software application. The treatment module 304 may receive a treatment version of a software application running on computing devices (e.g., the computing devices 110 of Fig. 1) is to be deployed in the treatment group for testing the software application, and the application distribution module 306 may, in response, use the application distribution module 306 to send the treatment version of the software application to the computing devices in the treatment group via one or more communication units 342. For example, in response to receiving a request from the testing module 304 to send the treatment version of the software application, the application distribution module 306 may generate a software update to update the software application to the treatment version of the software application and send the software update to update the software application to the treatment version of the software application to the computing devices in the treatment group.

[0100] Accordingly, when the testing module 304 sends the treatment version of the software application to the computing devices in the treatment group using the application distribution module 306, the testing module 304 may also use the application distribution module 306 to send one or more versions of the software application to computing devices in the control group for testing the software application. Specifically, the testing module 304 may determine the version of the software application installed on each computing device in the control group for testing the software application. If the testing module 304 determines that the version of the software application installed on the computing device in the control group is not the control version of the software application, the testing module 304 may send the control version of the software application to the computing device via the application distribution module 306.

[0101] Similarly, if the testing module 304 determines that the version of the software application installed on the computing device in the control group is the control version of the software application, the testing module 304 may send the treatment version of the software application to the computing device via the application distribution module 306. Sending the treatment version of the software application to a computing device in the control group that has the control version of the software application installed may trigger the computing device to reinstall the control version of the software application from a storage device of the computing devices.

[0102] As described above, the application distribution module 306 may send a version of the software application (e.g., the treatment version of the software application or the control version of the software application) in any form to one or more computing devices in the control group via one or more communication units 342 to send the treatment version of the software application to the computing devices in the treatment group via one or more communication units 342. For example, in response to receiving a request from the testing module 304 to send a particular version of the software application to a computing device, the application distribution module 306 may generate a software update to update the software application to the particular version of the software application and send the software update to update the software application to the particular version of the software application to the computing device.

[0103] Fig. 4A-4B are block diagrams illustrating software testing according to aspects of this disclosure. As in Fig. 4A, the computing system 402, which is an example of the computing system 102 of Fig. 1, with the computing devices 410A and 410B, which are examples of the computing devices 110 of Fig. 1 to perform testing of the software application 422. The computing device 410A may be in a control group to perform a control treatment experiment on the software application 422, while the computing device 410B may be in a treatment group to perform a control release experiment.

[0104] The software application 422 may include modules A, B, C, and D, and the control version of the software application 422 may include version 2 of module A, version 2 of module B, version 1 of module C, and version 1 of module D. The computing devices 410A and 410B may each have the control version of the software application 422 installed. Thus, each of the software applications 422 installed on the computing devices 410A and 410B includes version 2 of module A, version 2 of module B, version 1 of module C, and version 1 of module D.

[0105] As part of conducting a control treatment experiment, the computing system 402 may deploy the treatment version of the software application 422 on computing devices in the treatment group for the control treatment experiment. While the control version of the software application 422 includes version 1 of module A, version 1 of module B, version 2 of module C, and version 2 of module D, the treatment version of the software application 422 may include version 3 of module A, version 2 of module B, version 3 of module C, and version 1 of module D. For example, upgrading the control version of the software application 422 to the treatment version of the software application 422 may include upgrading version 2 of module A in the control version of the software application 422 to version 3 of module A and upgrading version 1 of module C in the control version to version 3 of module C.

[0106] Therefore, computing system 402 may implement the treatment version of software application 422 in the form of a software update that updates version 2 of module A in the control version of software application 422 to version 3 of module A and updates version 1 of module C in the control version to version 3 of module C. In some examples, the software update may include code to install the full treatment version of software application 422. That is, the software update may install any of version 3 of module A, version 2 of module B, version 3 of module C, and version 1 of module D included in the treatment version of software application 422.

[0107] In some examples, the software update may update the control version of the software application to the treatment version of the software application 422 by including code to update less than the entire control version of the software application 422 to the treatment version of the software application 422. That is, the software update may instead update only the portions of the control version of the software application 422 that differ from the treatment version of the software application 422. Since in the example of Fig. 4A only modules A and C of the control version of the software application 422 differ from the treatment version of the software application 422, the software update can update modules A and C of the control version of the software application 422 to the treatment version of modules A and C.

[0108] Thus, computing system 402 may send software update 450B to computing device 410B for updating the control version of module A and module C of software application 422 to the treatment version of module A and module C (e.g., version 3 of module A and version 3 of module C). Software update 450B may include code for installing the treatment version of module A and module C, such as by replacing the control version of module A and module C with the treatment version of module A and module C, patching the control version of module A and module C to the treatment version of module A and module C, or otherwise updating the control version of module A and module C to the treatment version of module A and module C.

[0109] Computing device 410B, in response to receiving software update 450B, may apply software update 450B to update the control version of software application 422 to the treatment version of software application 422. That is, computing device 410B may replace the control version of module A and module C with the treatment version of module A and module C, patch the control version of module A and module C to the treatment version of module A and module C, or otherwise update the software update of module A and module C to the treatment version of module A and module C to result in the treatment version of software application 422 including version 3 of module A, version 2 of module B, version 3 of module C, and version 1 of module D being installed on computing device 410B.

[0110] Computing system 402 may accordingly send software update 450A to computing device 410A in the control group for the control treatment experiment. Computing system 402 may determine that the control version of software application 422 is installed on computing device 410A and, in response, may send software update 450A to update software application 422 to the treatment version of software application 422 to trigger computing device 410A to reinstall the control version of software application 422 from a storage device of computing device 410A.

[0111] In the example of Fig. 4A, the software update 450A sent to the computing device 410A includes code to update the same modules as the software update 450B sent to the computing device 410B. That is, the software update 450A includes code to install the treatment version of Module A and Module C, such as by replacing the control version of Module A and Module C with the treatment version of Module A and Module C, patching the control version of Module A and Module C to the treatment version of Module A and Module C, or otherwise updating the control version of Module A and Module C to the treatment version of Module A and Module C.

[0112] In response to receiving the software update 450A, the computing device 410A may determine to install the software update 450A or to reinstall the control version of the software application 422 from a storage device of the computing device 410A. For example, the computing device 410A may determine whether the currently installed version of the software application 422 is the control version of the software application 422, and may determine to reinstall the currently installed version of the software application 422 if the currently installed version of the software application 422 is the control version of the software application 422.

[0113] If the computing device 410A determines to reinstall the control version of the software application 422, the computing device 410A may reinstall the control version of the software application 422 from a storage device of the computing device 410A. That is, the computing device 410A may store data for reinstalling the control version of the software application 422, such as an installation package, an application package, etc., on the hard disk. Thus, in response to receiving the software update 450A, the computing device 410A may reinstall the control version of the software application 422 using the data for reinstalling the control version of the software application 422 stored in the disk storage (e.g., on the hard disk) of the computing device 410A.In some examples, the computing device 410A may also store the software update 450A in the storage device of the computing device 410A, which may be used by the computing device 410A at a later time to upgrade the control version of the software application 422 to the treatment version of the software application 422 without having to re-download a software update to upgrade the software application 422 to the treatment version of the software application 422.

[0114] In some examples, when computing system 402 determines to send a software update to update software applications 422 to the control version of software application 422, computing system 402 may generate a software update that includes code to update the same modules that are updated by a software update to update software applications 422 to the treatment version of software application 422.That is, even if the software update for updating software applications 422 to the control version of software applications 422 includes some, but not all, of the modules updated by a software update for updating software applications 422 to the treatment version of software applications 422, the software update for updating software applications 422 to the control version of software applications 422 may still include code for updating the same modules updated by a software update for updating software applications 422 to the treatment version of software applications 422.By including code to update the same modules that are updated by a software update to update the software application 422 to the treatment version of the software application 422, the software update to update the software application 422 to the control version of the software application 422 may be closer in size to the software update to update the software application 422 to the treatment version of the software application 422, thereby reducing the amounts of bias introduced by size differences between these software updates when comparing performance data associated with downloads of these software updates.

[0115] As in Fig. 4B, the computing system 402, which is an example of the computing system 102 of Fig. 1, with the computing devices 410C and 410D, which are examples of the computing devices 110 of Fig. 1 to perform testing of the software application 422. The computing device 410C may be in a control group to perform a control treatment experiment on the software application 422, while the computing device 410D may be in a treatment group to perform a control release experiment.

[0116] The software application 422 may include modules A, B, C, and D, and the control version of the software application 422 may include version 2 of module A, version 2 of module B, version 1 of module C, and version 1 of module D. However, neither the computing device 410C nor the computing device 410D has the control version of the software application 422 installed. Instead, each of the computing devices 410C and 410B may have an older version of the software application 422 installed that includes version 1 of module A, version 1 of module B, version 1 of module C, and version 1 of module D.

[0117] As part of conducting a control treatment experiment, the computing system 402 may deploy the treatment version of the software application 422 on computing devices in the treatment group for the control treatment experiment. While the version of the software application 422 installed on the computing device 410D includes version 1 of module A, version 1 of module B, version 1 of module C, and version 1 of module D, the treatment version of the software application 422 may include version 3 of module A, version 2 of module B, version 3 of module C, and version 1 of module D. For example,upgrading the control version of the software application 422 to the treatment version of the software application 422 may include upgrading version 1 of module A in the control version of the software application 422 to version 3 of module A, upgrading version 1 of module B to version 2 of module B, and upgrading version 1 of module C in the control version to version 3 of module C.

[0118] Computing system 402 may therefore deploy the treatment version of software application 422 in the form of a software update 450D on computing device 410D that updates version 1 of module A in the control version of software application 422 to version 3 of module A, that updates version 1 of module B to version 2 of module B, and that updates version 1 of module C in the control version to version 3 of module C. In some examples, the software update may include code to install the full treatment version of software application 422. That is, the software update may install any of version 3 of module A, version 2 of module B, version 3 of module C, and version 1 of module D included in the treatment version of software application 422.In response to receiving a software update 450D, the computing device 410B may apply the software update 450D to update the version of the software application 422 installed on the computing device 410B to the treatment version of the software application 422.

[0119] Computing system 402 may accordingly send software update 450C to computing device 410C in the control group for the control treatment experiment. Computing system 402 may determine that the version of software application 422 installed on computing device 410C is not the control version of software application 422 and may then send software update 450C to computing device 410C to update software application 422 to the control version of software application 422.

[0120] The control version of the software application 422 includes version 2 of module A, version 2 of module B, version 1 of module C, and version 1 of module D. Thus, a software update for updating the software application 422 to the control version of the software application 422 may include at least code for updating version 1 of module A in the control version of the software application 422 to version 2 of module A and for updating version 1 of module B to version 2 of module B.

[0121] However, unlike update 450D, a software update that only includes code to update version 1 of module A in software application 422 to version 2 of module A and to update version 1 of module B in software application 422 to version 2 of module B does not include the code to update module C. Thus, computing system 402 may send to computing device 410C a software update 450C that includes code to update version 1 of module A in software application 422 to version 2 of module A and to update version 1 of module B in software application 422 to version 2 of module B, as well as code to update module C in software application 422 to version 3 of module C.

[0122] In response to receiving the software update 450C, the computing device 410C may determine to install the software update 450C or to reinstall the currently installed version of the software application 422 from a storage device of the computing device 410C. For example, the computing device 410C may determine whether the currently installed version of the software application 422 is the control version of the software application 422, and may determine to reinstall the currently installed version of the software application 422 if the currently installed version of the software application 422 is the control version of the software application 422.

[0123] In the example of Fig. 4B, the computing device 410C may determine that the currently installed version of the software application 422 is not the control version of the software application 422, and may therefore install the software update 450C to update the version of the software application 422 installed on the computing device 410C to the control version of the software application 422. In particular, the computing device 410C may deploy the software update 450C to update version 1 of module A in the software application 422 to version 2 of module A and to update version 1 of module B to version 2 of module B.Furthermore, since the control version of the software application 422 does not include updated versions of modules C and D, the computing device 410C may also reinstall version 1 of module C and version 1 of module D from the storage device of the computing device 410C to update the software application 422 to the control version of the software application 422. In this way, the computing device 410C may update an older version of the software application 422 to the control version of the software application 422.

[0124] Fig. 5 is a flowchart illustrating example operations performed by an example computing system configured to perform software testing in accordance with one or more aspects of the present disclosure. Fig. 5 is described below in the context of the environment 100 of Fig. 1 and the computing system 302 of Fig. 3 described.

[0125] As in Fig. 5, one or more processors 340 of computer system 302 may determine that a software application 122 installed on a computing device 110-1A in a control group 140 associated with a control treatment experiment for the software application 122 is a control version of the software application 122 (502). Based at least in part on the computing device 110-1A being in the control group 140 and the software application 122 installed on the computing device 110-1A being the control version of the software application 122, the one or more processors 340 may determine a software update to update the software application 122 to a treatment version of the software application 122 to be sent to the computing device 110-1A (504).The one or more processors 340 may send the software update to the computing device 110-1A to trigger the computing device 110-1A to reinstall the control version of the software application 122 on the computing device 110-1A from a storage device of the computing device 110-1A in response to receiving the software update (506).

[0126] Fig. 6 is a flowchart illustrating example operations performed by an example computing device located in a control group for a control treatment experiment to test a software application, according to one or more aspects of the present disclosure. Fig. 6 is described below in the context of the environment 100 of Fig. 1 and the computing device 210 of Fig. 2 described.

[0127] As in Fig.As shown in Figure 6, one or more processors 240 of computing device 210 may receive a software update from computing system 102 for updating a software application 222, wherein computing device 210 is in a control group associated with a control treatment experiment for software application 222, and wherein a control version of software application 222 is installed on the computing device (602). The one or more processors 240 may determine that the software update is for updating software application 222 to a treatment version of software application 222 (604).The one or more processors 240, in response to determining that the software update is to update the software application 222 to the treatment version of the software application 222, may reinstall the control version of the software application 222 on the computing device 210 from a storage device 248 of the computing device 210 (606).

[0128] Aspects of this disclosure include the following examples.

[0129] Example 1: A method includes determining, by one or more processors of a computing system, that a software application installed on a computing device in a control group associated with a control treatment experiment for the software application is a control version of the software application; determining, by the one or more processors and based at least in part on the computing device being in the control group and the software application installed on the computing device being the control version of the software application, a software update to update the software application to a treatment version of the software application to be sent to the computing device;and sending, by the one or more processors, to the computing device the software update to trigger the computing device to reinstall the control version of the software application on the computing device from a storage device of the computing device in response to receiving the software update;

[0130] Example 2: The method of Example 1 further includes sending, by the one or more processors, the software update to update the software application to the treatment version of the software application to a plurality of computing devices in a treatment group associated with the control treatment experiment.

[0131] Example 3: The method of any one of examples 1 and 2, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the treatment version of the software application.

[0132] Example 4: The method of any of Examples 1-3 further includes determining, by one or more processors, whether the software application installed on a second computing device in the control group is the control version of the software application; and in response to determining that the software application installed on a second computing device is not the control version of the software application, sending, by the one or more processors, to the second computing device a second software update to update the software application to the control version of the software application.

[0133] Example 5: The method of any of Examples 1-4 further includes determining, by the one or more processors, the control version of the software application as one of: a version of the software application that is fully released to computing devices, or a most recent version of the software application prior to deployment of the treatment version of the software application to the plurality of computing devices in the treatment group.

[0134] Example 6: The method of any of Examples 1-5, wherein sending the software update further comprises: receiving, by the one or more processors and from the computing device, an automatic update request associated with the software application; and in response to receiving the automatic update request from the computing device, sending, by the one or more processors and to the computing device, the software update to trigger the computing device to reinstall the control version of the software application on the computing device from the storage device of the computing device in response to receiving the software update.

[0135] Example 7: The method of any of Examples 1-6, wherein sending the software update to the computing device further triggers the computing device to store a copy of the treatment version of the software application in the storage device of the computing device, and the method further includes sending, by the one or more processors, to the computing device an indication that the treatment version of the software application is released as the next official version of the software application, causing the computing device to install the copy of the treatment version of the software application from the computing device to replace the control version of the software application.

[0136] Example 8: A computing system including memory storing instructions; and one or more processors executing the instructions to: determine that a software application installed on a computing device in a control group associated with a control treatment experiment for the software application is a control version of the software application; determine a software update to update the software application to a treatment version of the software application to be sent to the computing device based at least in part on the computing device being in the control group and the software application installed on the computing device being the control version of the software application;and sending the software update to the computing device to trigger the computing device to reinstall the control version of the software application on the computing device from a storage device of the computing device in response to receiving the software update;

[0137] Example 9: The computing system of Example 8, wherein the one or more processors further execute instructions to: send the software update to update the software application to a treatment version of the software application to a plurality of computing devices in a treatment group associated with the control treatment experiment for the software application.

[0138] Example 10: The computing system of any one of examples 8 and 9, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the treatment version of the software application.

[0139] Example 11: The computing system of any of Examples 8-10, wherein the one or more processors further execute instructions to: determine that the software application installed on a second computing device in the control group is the control version of the software application; and in response to determining that the software application installed on a second computing device is not the control version of the software application, send a second software update to update the software application to the control version of the software application to the second computing device.

[0140] Example 12: The computing system of any of examples 8-11, wherein the one or more processors further execute instructions to: determine the control version of the software application as one of: a version of the software application that is fully released to computing devices, or a most recent version of the software application prior to deployment of the treatment version of the software application for the plurality of computing devices in the treatment group.

[0141] Example 13: The computing system of any of Examples 8-12, wherein to send the software update, the one or more processors further execute instructions to: receive an automatic update request associated with the software application and from the computing device; and in response to receiving the automatic update request from the computing device, send the software update to the computing device to trigger the computing device to reinstall the control version of the software application on the computing device from the storage device of the computing device in response to receiving the software update.

[0142] Example 14: The computing system of any of Examples 8-13, wherein the one or more processors further execute the instructions to send the software update to the computing device to further trigger the computing device to store a copy of the treatment version of the software application in the storage device of the computing device, further including sending, by the one or more processors, to the computing device an indication that the treatment version of the software application is released as the next official version of the software application, causing the computing device to install the copy of the treatment version of the software application from the computing device to replace the control version of the software application.

[0143] Example 15: A non-transitory computer-readable storage medium includes instructions that, when executed by one or more processors of a computing system, cause the one or more processors to: determine that a software application installed on a computing device in a control group associated with a control treatment experiment for the software application is a control version of the software application; determine a software update to update the software application to a treatment version of the software application to be sent to the computing device based at least in part on the computing device being in the control group and the software application installed on the computing device being the control version of the software application;and sending the software update to the computing device to trigger the computing device to reinstall the control version of the software application on the computing device from a storage device of the computing device in response to receiving the software update;

[0144] Example 16: The non-transitory computer-readable storage medium of Example 15, wherein the instructions further cause the one or more processors to: send the software update to update the software application to the treatment version of the software application to a plurality of computing devices in a treatment group associated with the control treatment experiment.

[0145] Example 17: The non-transitory computer-readable storage medium of any one of examples 15 and 16, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the treatment version of the software application.

[0146] Example 18: The non-transitory computer-readable storage medium of any of Examples 15-18, wherein the instructions further cause the one or more processors to: determine that the software application installed on a second computing device in the control group is not the control version of the software application; and in response to the second computing device being in the control group and the software application installed on a second computing device not being the control version of the software application, send a second software update to update the software application to the control version of the software application to the second computing device.

[0147] Example 19: The non-transitory computer-readable storage medium of any of Examples 15-18, wherein the instructions further cause the one or more processors to: determine the control version of the software application as one of: a version of the software application that is fully released to computing devices, or a most recent version of the software application prior to deployment of the treatment version of the software application to the plurality of computing devices in the treatment group.

[0148] Example 20: The non-transitory computer-readable storage medium of any of Examples 15-19, wherein the instructions that cause the one or more processors to send the software update further cause the one or more processors to: receive an automatic update request associated with the software application from the computing device; and in response to receiving the automatic update request from the computing device, send the software update to the computing device to trigger the computing device to reinstall the control version of the software application on the computing device from the storage device of the computing device in response to receiving the software update.

[0149] Example 21: A method includes receiving, by one or more processors of a computing device and from a computing system, a software update to update a software application, wherein the computing device is in a control group associated with a control treatment experiment for the software application, and wherein a control version of the software application is installed on the computing device; determining, by the one or more processors, whether the software update is to update the software application to a treatment version of the software application; and in response to determining that the software update is to update the software application to the treatment version of the software application, reinstalling, by the one or more processors, the control version of the software application on the computing device from a storage device of the computing device.

[0150] Example 22: The method of Example 21, wherein a software installation package for installing the control version of the software application is stored on the storage device of the computing device, and wherein reinstalling the control version of the software application further comprises: reinstalling, by the one or more processors, the control version of the software application on the computing device using the software installation package.

[0151] Example 23: The method of any one of examples 21 and 22, wherein reinstalling the control version of the software application further includes updating, by the one or more processors and using the software update, a copy of the control version of the software application to the treatment version of the software application; and storing, by the one or more processors, a copy of the treatment version of the software application in the storage device of the computing device.

[0152] Example 24: The method of Example 23, wherein reinstalling the control version of the software application on the computing device further comprises: in response to determining that the computing device is in a test state, reinstalling, by the one or more processors, the control version of the software application on the computing device from the storage device of the computing device.

[0153] Example 25: The method of Example 24 further includes receiving, by the one or more processors and from the computing system, an indication that the computing device is in the test state.

[0154] Example 26: The method of Example 25, wherein receiving the indication that the computing device is in the test state further comprises: sending, by the one or more processors and to the computing system, a query regarding whether the computing device is in the test state; and in response to sending the query, receiving, by the one or more processors and from the computing system, the indication that the computing device is in the test state.

[0155] Example 27: The method of any of Examples 24-26 further includes determining, by the one or more processors, whether the computing device is in the test state; and in response to determining that the computing device is no longer in the test state, installing, by the one or more processors, the copy of the treatment version of the software application stored on the computing device to replace the control version of the software application.

[0156] Example 28: The method of any of examples 21-27, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the treatment version of the software application.

[0157] Example 29: The method of any of examples 21-28, wherein receiving the software update is responsive to the one or more processors sending an automatic update request for the software application to the computing system.

[0158] Example 30: A computing device includes a memory storing instructions; a storage device; and one or more processors executing the instructions to: receive a software update from a computing system to update a software application, wherein the computing device is in a control group associated with a control treatment experiment for the software application, and wherein a control version of the software application is installed on the computing device; determine whether the software update is to update the software application to a treatment version of the software application; and in response to determining that the software update is to update the software application to the treatment version of the software application, reinstall the control version of the software application on the computing device from the storage device of the computing device.

[0159] Example 31: The computing device of Example 30, wherein the storage device of the computing device is configured to store a software installation package for installing the control version of the software application, and wherein to reinstall the control version of the software application, the one or more processors are further configured to: reinstall the control version of the software application on the computing device using the software installation package.

[0160] Example 32: The computing device of any of examples 30 and 31, wherein to reinstall the control version of the software application, the one or more processors further execute instructions to: update a copy of the control version of the software application to the treatment version of the software application using the software update; and store a copy of the treatment version of the software application in the storage device of the computing device.

[0161] Example 33: The computing device of any one of Example 32, wherein reinstalling the control version of the software application on the computing device further comprises: in response to determining that the computing device is in a test state, reinstalling the control version of the software application on the computing device from the storage device of the computing device.

[0162] Example 34: The computing device of Example 33, wherein the one or more processors are further configured to: receive an indication from the computing system that the computing device is in the test state.

[0163] Example 35: The computing device of Example 34, wherein to receive the indication that the computing device is in the test state, the one or more processors are further configured to: send a query to the computing system regarding whether the computing device is in the test state; and in response to sending the query, receive the indication that the computing device is in the test state from the computing system.

[0164] Example 36: The computing device of any of examples 33-35, wherein the one or more processors are further configured to: determine whether the computing device is in the test state; and in response to determining that the computing device is no longer in the test state, reinstall the copy of the treatment version of the software application stored on the computing device to replace the control version of the software application.

[0165] Example 37: The computing device of any of examples 30-36, wherein the software update to update the software application includes code to update the software application to the control version of the software application and code to update the software application to the treatment version of the software application.

[0166] Example 38: The computing device of any of examples 30-37, wherein the one or more processors are configured to receive the software update in response to sending an automatic update request for the software application to the computing system.

[0167] Example 39: A non-transitory computer-readable storage medium including instructions that, when executed by one or more processors of a computing device, cause the one or more processors to receive, from a computing system, a software update for updating a software application, wherein the computing device is in a control group associated with a control treatment experiment for the software application, and wherein a control version of the software application is installed on the computing device; determining whether the software update is for updating the software application to a treatment version of the software application;and in response to determining that the software update is for updating the software application to the treatment version of the software application, reinstalling the control version of the software application on the computing device from a storage device of the computing device;

[0168] Example 40: The non-transitory computer-readable storage medium of Example 39, wherein the instructions that cause the one or more processors to reinstall the control version of the software application further cause the one or more processors to: update a copy of the control version of the software application to the treatment version of the software application using the software update; and store a copy of the treatment version of the software application in the storage device of the computing device.

[0169] By way of example, but not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other storage media that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, each combination is properly referred to as a computer-readable medium.For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair cable, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. However, it is understood that computer-readable storage media and media, as well as data storage media, do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transitory, tangible storage media.In this context, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray Disc. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the foregoing are also intended to be included within the scope of a computer-readable medium.

[0170] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" as used herein may refer to any of the foregoing structures or any other structures suitable for implementing the techniques described in this document. Furthermore, the functionality described in this document may, in some aspects, be provided in dedicated hardware and / or software modules. Furthermore, the techniques could be implemented entirely in one or more circuits or logic elements.

[0171] The techniques of this disclosure may be implemented in a wide variety of devices or devices, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to highlight functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Rather, as described above, various units may be combined into one hardware unit or provided by a collection of interoperable hardware units, including one or more processors, as described above in conjunction with suitable software and / or firmware.

[0172] Various embodiments have been described. These and other embodiments are within the scope of the following claims.

Claims

[1] Method comprising: determining, by one or more processors of a computing system, that a software application installed on a computing device in a control group associated with a control treatment experiment for the software application is a control version of the software application; determining, by the one or more processors, a software update to update the software application to a treatment version of the software application to be sent to the computing device, based at least in part on the computing device being in the control group and the software application installed on the computing device being the control version of the software application; and Sending, by the one or more processors, the software update to trigger the computing device to reinstall the control version of the software application on the computing device from a storage device of the computing device in response to receiving the software update, to the computing device. [2] The method of claim 1, further comprising: Sending the software update for updating the software application to the treatment version of the software application, by the one or more processors, to a plurality of computing devices in a treatment group associated with the control treatment experiment for the software application. [3] A method according to any one of claims 1 and 2, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the treatment version of the software application. [4] Method according to any one of claims 1-3, further comprising: Determining, by the one or more processors, whether the software application installed on a second computing device in the control group is the control version of the software application; and in response to determining that the software application installed on a second computing device is not the control version of the software application, sending, by the one or more processors, to the second computing device a second software update to update the software application to the control version of the software application. [5] Method according to any one of claims 1-4, further comprising: determining, by the one or more processors, the control version of the software application as one of: a version of the software application that is fully released to computing devices, or a most recent version of the software application prior to the treatment version of the software application. [6] The method of any of claims 1-5, wherein sending the software update further comprises: Receiving an automatic update request associated with the software application by the one or more processors and from the computing device; and in response to receiving the automatic update request from the computing device, sending, by the one or more processors, to the computing device the software update to trigger the computing device to reinstall the control version of the software application on the computing device from the storage device of the computing device in response to receiving the software update. [7] The method of any of claims 1-6, wherein sending the software update to the computing device further triggers the computing device to store a copy of the treatment version of the software application in the storage device of the computing device, and further comprising: Sending, by the one or more processors, to the computing device an indication that the treatment version of the software application is released as the next official version of the software application, causing the computing device to install the copy of the treatment version of the software application stored on the computing device to replace the control version of the software application. [8] Computing system comprising: a memory that stores instructions; and one or more processors executing the instructions to perform the method of any of claims 1-7. [9] A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform the method of any one of claims 1-7. [10] Method comprising: Receiving a software update for updating a software application by one or more processors of a computing device and from a computing system, wherein the computing device is in a control group associated with a control treatment experiment for the software application, and wherein a control version of the software application is installed on the computing device; Determining, by the one or more processors, whether the software update is for updating the software application to a treatment version of the software application; and in response to determining that the software update is to update the software application to the treatment version of the software application, reinstalling, by the one or more processors, the control version of the software application on the computing device from a storage device of the computing device. [11] The method of claim 10, wherein a software installation package for installing the control version of the software application is stored on the storage device of the computing device, and wherein reinstalling the control version of the software application further comprises: reinstalling, by the one or more processors, the control version of the software application on the computing device using the software installation package. [12] The method of any of claims 10 and 11, wherein reinstalling the control version of the software application further comprises: Updating a copy of the control version of the software application to the treatment version of the software application by the one or more processors and using the software update; and Storing, by the one or more processors, a copy of the treatment version of the software application in the storage device of the computing device. [13] The method of claim 12, wherein reinstalling the control version of the software application on the computing device further comprises: in response to determining that the computing device is in a test state, reinstalling, by the one or more processors, the control version of the software application on the computing device from the storage device of the computing device. [14] The method of claim 13, further comprising: Receiving, by the one or more processors and from the computing system, an indication that the computing device is in the test state. [15] The method of claim 14, wherein receiving the indication that the computing device is in the test state further comprises: Sending, by the one or more processors and to the computing system, a query regarding whether the computing device is in the test state; and in response to sending the query, receiving, by the one or more processors and from the computing system, the indication that the computing device is in the test state. [16] A method according to any one of claims 13-15, further comprising: Determining, by the one or more processors, whether the computing device is in a test state; and in response to determining that the computing device is no longer in a test state, installing, by the one or more processors, the copy of the treatment version of the software application stored on the computing device to replace the control version of the software application. [17] The method of any of claims 10-16, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the treatment version of the software application. [18] The method of any of claims 10-17, wherein receiving the software update is responsive to the one or more processors sending an automatic update request for the software application to the computing system. [19] Computing system comprising: a memory that stores instructions; and one or more processors executing the instructions to perform the method of any of claims 10-18. [20] A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a computing device, cause the one or more processors to perform the method of any one of claims 10-18.

Images