Balanced control-processing experiments for software testing

Abstract

A computing system may determine that a software application installed at a computing device in a control group associated with a control-processing experiment for the software application is a controlled version of the software application. The computing system may determine a software update to be sent to the computing device for updating the software application based at least in part on the computing device being in the control group and the software application installed at the computing device being a control version of the software application. The computing system may send a software update to the computing device to trigger the computing device to reinstall a control version of the software application at the computing device from a storage device of the computing device in response to receiving the software update.

Description

Background Art

[0001] A software developer may perform a control-treatment experiment (i.e., A / B testing) to evaluate whether performance issues experienced during downloading, installing, and / or executing a new version of a software application are introduced by the new version of the software application. The software developer may collect performance data for the new version of the software application and performance data for an old version of the software application, and may use the collected performance data to identify performance issues introduced by the new version of the software application. Summary of the invention

[0002] Generally speaking, the techniques of the present disclosure relate to performing control-treatment experiments for software applications in a manner that improves the performance of such experiments. A computing system can deploy a treatment version of the software application at computing devices in a treatment group, and the performance of the treatment version of the software application can be compared to the performance of a control version of the software application installed at computing devices in a control group.

[0003] The computing system may collect performance data regarding downloading and installation of a processed version of a software application at a computing device. To generate corresponding performance data from computing devices in a control group, the computing device may send the processed version of the software application to computing devices in the control group that have installed the control version of the software application. The computing devices in the control group may reinstall the control version of the software application in response to receiving the processed version of the software application.

[0004] Because both computing devices in the control group and computing devices in the processing group can download processed versions of software applications, the techniques of the present disclosure can reduce any bias in comparing performance data about computing devices in the control group and performance data about computing devices in the processing group, which may be introduced by computing devices in the control group and computing devices in the processing group downloading different data with very different characteristics (e.g., different download sizes). In addition, by causing computing devices in the control group to reinstall the controlled version of the software application, the techniques of the present disclosure can trigger the computing devices to perform the installation action, thereby enabling performance data about the installation of the processed version of the software application at computing devices in the processing group to be compared with corresponding performance data about the reinstallation of the controlled version of the software application at computing devices in the control group. In this way, the techniques of the present disclosure improve the technical field of software testing.

[0005] In addition, since the computing devices in the control group download the processed version of the software application, when the processed version of the software application becomes a new release version of the software application, the computing devices in the control group may not have to re-download the processed version of the software application to upgrade to the new release version of the software application. In this way, the technology of the present disclosure can also reduce the network usage by the computing devices in the control group, thereby potentially providing further technical improvements.

[0006] In some aspects, the technology described herein relates to a method comprising: determining, by one or more processors of a computing system, a software application installed at a computing device in a control group associated with a control-treatment experiment for the software application is a control version of the software application; determining, by the one or more processors, a software update to be sent to the computing device for updating the software application to a treatment version of the software application based at least in part on the computing device being in the control group and the software application installed at the computing device being the control version of the software application; and sending, by the one or more processors, the software update to the computing device to trigger the computing device to reinstall the control version of the software application at the computing device from a storage device of the computing device in response to receiving the software update.

[0007] In some aspects, the technology described herein relates to a computing system comprising: a memory that stores instructions; and one or more processors that execute the instructions to: determine that a software application installed at a computing device in a control group associated with a control-treatment experiment for the software application is a control version of the software application; determine a software update to be sent to the computing device for updating the software application to a treatment version of the software application based at least in part on the computing device being in the control group and the software application installed at the computing device being the control version of the software application; and send the software update to the computing device to trigger the computing device to reinstall the control version of the software application at the computing device from a storage device of the computing device in response to receiving the software update.

[0008] In some aspects, the technology described herein relates to a non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors, cause one or more processors of a computing system to: determine that a software application installed at a computing device in a control group associated with a control-treatment experiment for the software application is a control version of the software application; determine a software update to be sent to the computing device for updating the software application to a treatment version of the software application based at least in part on the computing device being in the control group and the software application installed at the computing device being the control version of the software application; and send the software update to the computing device to trigger the computing device to reinstall the control version of the software application at the computing device from a storage device of the computing device in response to receiving the software update.

[0009] In some aspects, the technology described herein relates to an apparatus comprising: means for determining that a software application installed at a computing device in a control group associated with a control-treatment experiment for the software application is a control version of the software application; means for determining a software update to be sent to the computing device for updating the software application to a treatment version of the software application based at least in part on the computing device being in the control group and the software application installed at the computing device being the control version of the software application; and means for sending the software update to the computing device to trigger the computing device to reinstall the control version of the software application at the computing device from a storage device of the computing device in response to receiving the software update.

[0010] In some aspects, the technology described herein relates to a method comprising: receiving, by one or more processors of a computing device, a software update for updating a software application from a computing system, wherein the computing device is in a control group associated with a control-treatment experiment for the software application, and wherein a control version of the software application is installed at the computing device; determining, by the one or more processors, whether the software update is for updating the software application to a treatment version of the software application; and in response to determining that the software update is for updating the software application to the treatment version of the software application, reinstalling, by the one or more processors, the control version of the software application from a storage device of the computing device at the computing device.

[0011] In some aspects, the technology described herein relates to a computing system comprising: a memory that stores instructions; a storage device; and one or more processors that execute instructions to: receive a software update for updating a software application from the computing system, wherein the computing device is in a control group associated with a control-treatment experiment for the software application, and wherein a control version of the software application is installed at the computing device; determine whether the software update is for updating the software application to a treatment version of the software application; and in response to determining that the software update is for updating the software application to the treatment version of the software application, reinstall the control version of the software application from the storage device of the computing device at the computing device.

[0012] In some aspects, the technology described herein relates to a non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors, cause one or more processors of a computing device to: receive a software update for updating a software application from a computing system, wherein the computing device is in a control group associated with a control-treatment experiment for the software application, and wherein a control version of the software application is installed at the computing device; determine whether the software update is for updating the software application to a treatment version of the software application; and reinstall the control version of the software application from a storage device of the computing device at the computing device in response to determining that the software update is for updating the software application to the treatment version of the software application.

[0013] In some aspects, the technology described herein relates to an apparatus comprising: means for receiving a software update for updating a software application from a computing system, wherein the computing device is in a control group associated with a control-treatment experiment for the software application, and wherein a control version of the software application is installed at the computing device; means for determining whether the software update is for updating the software application to the treatment version of the software application; and means for reinstalling the control version of the software application from a storage device of the computing device at the computing device in response to determining that the software update is for updating the software application to the treatment version of the software application.

[0014] The details of one or more examples are set forth in the accompanying drawings and the description below.Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a conceptual diagram illustrating an example environment for software testing according to one or more aspects of the present disclosure.

[0016] Figure 2is a block diagram illustrating additional details of an example computing device according to one or more aspects of the present disclosure.

[0017] Figure 3 is a block diagram illustrating an example computing system in accordance with one or more aspects of the present disclosure.

[0018] FIG. 4A to FIG. 4B is a block diagram illustrating software testing according to aspects of the present disclosure.

[0019] Figure 5 is a flow diagram illustrating example operations performed by an example computing system configured to perform software testing in accordance with one or more aspects of the present disclosure.

[0020] Figure 6 is a flow diagram illustrating example operations performed by an example computing device in a control group of a control-process experiment for testing a software application in accordance with one or more aspects of the present disclosure. DETAILED DESCRIPTION

[0021] Figure 1 is a conceptual diagram illustrating an example environment for software testing according to one or more aspects of the present disclosure. Figure 1 In the example of , environment 100 may include a computing system 102 that communicates with computing devices 110 - 1A to 110 - 1N and computing devices 110 - 2A to 110 - 2N (collectively referred to as “computing devices 110 ”) via a network 130 .

[0022] Computing system 102 may represent any suitable computing system capable of sending and receiving information to and from a network (network 130), such as one or more desktop computers, laptops, mainframes, servers, cloud computing systems, etc. In some examples, computing system 102 may represent a cloud computing system that provides access to its corresponding services via a cloud.

[0023] The computing system 102 includes a testing module 104, an application distribution module 106, and a testing data store 108. The testing module 104 and the application distribution module 106 may perform the operations described herein using software, hardware, firmware, or a mixture of software, hardware, and firmware resident in and / or executed at the computing device 102. The computing system 102 may execute the testing module 104 and the application distribution module 106 using multiple processors or multiple devices, as a virtual machine executing on underlying hardware, as one or more services of an operating system or computing platform, and / or as one or more executable programs at the application layer of the computing platform of the computing system 102.

[0024] Testing module 104 may perform functions associated with executing control-process experiments (ie, A / B testing) for software applications installed at computing device 110. In some examples, testing module 104 may be included in an operating system of computing system 102.

[0025] Application distribution module 106 may perform functions associated with distributing software applications and / or software updates to computing device 110. For example, application distribution module 106 may send software updates for updating software applications (e.g., software application 122) to computing device 110. Furthermore, application distribution module 106 may handle automatic and / or manual updates of software applications from computing device 110 by sending updates to the software applications to computing device 110. In some examples, application distribution module 106 may be included in an operating system of computing system 102.

[0026] The test data store 108 may be any suitable data store for storing data associated with performing a control-process experiment for a software application, such as a database, a repository, etc. For example, the test data store 108 may store an indication of the version of the software application 122 installed at each of the computing devices 110, a test status of the control-process experiment, an indication of the process version of the software application 122, an indication of the control version of the software application 122, etc.

[0027] Network 130 represents any public or private communication network for transmitting data between computing systems, servers, and computing devices, such as cellular, Wi-Fi, and / or other types of networks. Network 130 may include one or more network hubs, network switches, network routers, or any other network devices that are operably coupled to each other to provide information exchange between computing system 102 and computing device 110. Computing device 110 and computing system 102 may send and receive data over network 130 using any suitable communication technology. Each of computing system 102 and computing device 110 is operably coupled to network 130 using a corresponding network link, such as Ethernet, Wi-Fi, or any other type of wired and / or wireless network connection.

[0028] Computing devices 110 each represent an individual mobile or non-mobile computing device. Examples of each of computing devices 110 include a mobile phone, a tablet computer, a laptop computer, a desktop computer, a server, a mainframe, a set-top box, a television, a wearable device (e.g., a computerized watch, computerized glasses, a computerized headset, a computerized glove, etc.), a home automation device or system (e.g., a smart thermostat or a home assistant device), a personal digital assistant (PDA), a gaming system, a media player, an e-book reader, a mobile television platform, a car navigation or infotainment system, or any other type of mobile, non-mobile, wearable, and non-wearable computing device.

[0029] Each of the computing devices 110 includes an application management module 116 and software applications 112. The application management module 116 and software applications 112 may perform the operations described herein using software, hardware, firmware, or a mixture of software, hardware, and / or firmware resident in and / or executed at the computing device 110-1A. Each of the computing devices 110 may execute the application management module 116 and software applications 112 using multiple processors or multiple devices, as a virtual machine executing on underlying hardware, as one or more services of an operating system or computing platform, and / or as one or more executable programs at the application layer of the computing platform of each of the computing devices 110.

[0030] Software application 122 may be any software application executable by computing device 110 to perform one or more functions. In some examples, software application 122 may be a first-party application provided by an operating system or by a manufacturer of computing device 110. In some examples, software application 122 may be an application provided by a third-party developer.

[0031] Application management module 116 may perform functions associated with managing software applications installed at computing device 110, including software applications 122. For example, application management module 116 may download (e.g., from computing system 102) and install software applications, reinstall applications already installed at computing device 110, perform manual and / or automatic updates of software applications, etc. In some examples, application management module 116 may be included in an operating system of computing device 110, or may be part of an app store application at computing device 110.

[0032] exist Figure 1In an example of the present invention, a developer may use environment 100 to perform a control-treatment experiment (i.e., A / B testing) to test a version of a software application (e.g., software application 122) before such version is potentially released. The developer may perform such a control-treatment experiment to evaluate whether performance issues experienced during downloading, installing, and / or executing a version of the software application are introduced by the version of the software application.

[0033] Developers of software applications may periodically release new versions of the software application to provide new features, fix bugs in previous versions of the software application, etc. However, in some cases, a new version of the software application may cause problems that may negatively affect the performance of a computing device that has the new version of the software application installed. For example, a computing device may fail to successfully download and / or install the new version of the software application, the new version of the software application may suffer from slower startup speeds, the new version of the software application may crash when used, the new version of the software application may introduce new bugs, etc.

[0034] A developer of a software application may perform controlled-processing experiments on a version of the software application to evaluate whether performance issues experienced during downloading, installing, and / or executing the version of the software application were introduced by the version of the software application. To perform such experiments, the developer may perform a controlled release of a new version of the software application to roll out the new version of the software application to a limited number of computing devices. That is, the developer may release the new version of the software application to fewer than all computing devices on which the software application is installed.

[0035] By performing a controlled release of a new version of a software application, a developer can divide the computing devices on which the software application is installed into two groups of equal or very similar size: a treatment group that includes computing devices on which the new version of the software application is installed, and a control group that includes computing devices on which the old version of the software application is installed. In the context of a control-treatment experiment, the new version of the software application being tested is referred to as the treatment version of the software application. Additionally, the control group can include computing devices on which the same old version of the software application is installed, and in the context of a control-treatment experiment, the old version of the software application is referred to herein as the control version of the software application.

[0036] A developer of a software application may collect performance data associated with the software application from computing devices in a processing group on which a processed version of the software application is installed and computing devices in a control group on which a controlled version of the software application is installed. Such performance data may include download failure rates, installation failure rates, application crash rates, application launch delays, power usage, memory usage, frame rate metrics, or any other suitable performance data and / or metrics. The developer may compare the performance data collected from the computing devices in the processing group with the performance data collected from the computing devices in the control group to compare performance issues associated with the processed version of the computing devices with performance issues associated with the controlled version of the software application. In this way, the developer is able to determine performance issues associated with a new version of the software application and assess whether such performance issues are introduced by the new version of the software application.

[0037] Performance issues associated with the release of a new version of a software application may not only arise during startup or during execution of the software application. Instead, the release of a new version of a software application may sometimes introduce performance issues during the download and installation of the software application. Therefore, the developer may also collect performance data associated with the download and installation of a process version of the software application by computing devices in the process group.

[0038] In order to compare performance data associated with downloading and installing the processed version of the software application at computing devices in the treatment group with corresponding data generated from computing devices in the treatment group, the developer may generate a placebo version of the software application, and computing devices in the control group may download and install the placebo version of the software application, which may then become the control version of the software application. Thus, the developer is able to collect performance data associated with downloading and installing the placebo version of the software application by computing devices in the control group to compare with performance data associated with downloading and installing the processed version of the software application by computing devices in the treatment group.

[0039] However, using such a placebo version of a software application in order to generate performance data for a control group to compare with the performance data for the treatment group may result in certain technical problems. For example, the computing device may not know that the placebo version of the software application is an upgrade, and therefore may fail to install the placebo version of the software application. This may require the developer to create a build of the placebo version of the software application with a very specific version code so that the computing device knows that the placebo version of the software application is an upgrade.

[0040] Additionally, failure to upgrade to the placebo version of the software application may result in the computing devices in the control group running different versions of the software application, thereby creating an imbalance between the number of computing devices running the treatment version of the software application and the number of computing devices in the control group running the placebo version of the software application. This imbalance (also known as arm imbalance) may skew the comparison of performance data between the treatment group and the control group.

[0041] Additionally, a software update used to update a software application to a placebo version of the software application may have a different size than a software update used to update the software application to a processed version of the software application. The different sizes of the software updates may introduce differences in the download and installation performance of the placebo version of the software application and the processed version of the software application, thereby potentially skewing or biasing a comparison of performance data associated with downloading and installing the placebo version of the software application and performance data associated with downloading and installing the processed version of the software application.

[0042] According to aspects of the present disclosure, a computing system may use a version of a software application as a placebo when performing a control-treatment experiment for the software application in order to compare performance data associated with a control group with performance data associated with a treatment group. That is, in addition to deploying the treated version of the software application to computing devices in the treatment group, the computing system may also send the version of the software application to computing devices in the control group in order to collect and compare performance data about the treated version of the software application deployed at the treatment group and the control version of the software application deployed at the control group.

[0043] Specifically, the computing system may determine the version of the software application installed at the computing device for the computing devices in the control group. If the computing system determines that the version of the software application installed at the computing device is a control version of the software application, the computing system may send a processed version of the computing device to the computing device to trigger the computing device to reinstall the control version of the computing device. The computing device may reinstall the control version of the software application from a storage device of the computing device in response to receiving the processed version of the software application.

[0044] Because both computing devices in the control group and computing devices in the processing group can download processed versions of software applications, the techniques of the present disclosure can reduce any bias in comparing performance data about computing devices in the control group and performance data about computing devices in the processing group, which may be introduced by computing devices in the control group and computing devices in the processing group downloading different data with very different characteristics (e.g., different download sizes). In addition, by causing computing devices in the control group to reinstall the controlled version of the software application, the techniques of the present disclosure can trigger the computing devices to perform the installation action, thereby enabling performance data about the installation of the processed version of the software application at computing devices in the processing group to be compared with corresponding performance data about the reinstallation of the controlled version of the software application at computing devices in the control group. In this way, the techniques of the present disclosure improve the technical field of software testing.

[0045] Similarly, if the computing system determines that the version of the software application installed at the computing device is not the control version of the software application, the computing system may send the control version of the computing device to the computing device to trigger the computing device to install the control version of the computing device. The computing device may install the control version of the software application in response to receiving the control version of the software application. The control version of the software application is sent to the computing device in the control group that has not yet installed the control version of the software application and the control version is installed. In this way, the technology of the present disclosure can increase the number of computing devices in the control group on which the control version of the software application is installed, thereby improving the security of the computing devices in the control group by increasing the number of computing devices upgraded from the old version of the software application to the control version of the software application. In addition, increasing the number of computing devices in the control group on which the control version of the software application is installed can reduce the difference between the number of computing devices in the processing group on which the processing version of the software application is installed and the number of computing devices in the control group on which the control version of the software application is installed, thereby improving the technical field of software testing.

[0046] exist Figure 1 In the example of FIG. 1 , to perform such testing on software application 122, the developer may divide computing devices 110 used to perform such testing into two groups: control group 140 and processing group 142. Computing devices 110-1A through 110-1N (“computing device 110-1”) are included in control group 140, while computing devices 110-2A through 110-2M (“computing device 110-2”) are included in processing group 142.

[0047] The computing device 110-2 in the processing group 142 is a computing device for receiving and installing a new version of the software application 122 for testing, while the computing device 110-1 in the control group 140 may execute the control version of the software application 122. The developer may compare the performance data associated with the new version of the software application 122 collected from the computing device 110-2 in the processing group 142 with the performance data associated with the control version of the software application 122 collected from the computing device 110-1 in the control group 140 to determine performance issues associated with the new version of the software application 122 and to evaluate whether such performance issues are introduced by the new version of the software application 122.

[0048] The developer may select computing devices 110-1 to be included in control group 140 and computing devices 110-2 to be included in processing group 142 from computing devices 110 in any manner. In some examples, the developer may select computing devices 110-1 to be included in control group 140 and computing devices 110-2 to be included in processing group 142 such that control group 140 and processing group 142 include the same number of computing devices.

[0049] In some examples, computing system 102 may use one or more heuristics, such as implemented by one or more neural networks, to select computing device 110-1 from computing devices 110 to be included in control group 140 and to select computing device 110-2 from computing devices 110 to be included in processing group 142. The one or more neural networks may be implemented by computing system 102 or may be implemented by a remote computing system in communication with computing system 102.

[0050] In general, the one or more neural networks may include a plurality of interconnected nodes, and each node may apply one or more functions to a set of input values ​​corresponding to one or more characteristics and provide one or more corresponding output values. In the context of selecting computing devices 110 to be included in control group 140 and / or treatment group 142, the one or more characteristics may include usage data of software applications 122 at computing devices 110, usage data of computing devices 110, performance data collected from computing devices 110 during previous control-treatment experiments, and / or any other data about computing devices 110, and for each of computing devices 110, one or more corresponding output values ​​of the one or more neural networks may be an indication of whether the computing device is included in control group 140, whether the computing device is included in treatment group 142, or whether the computing device is not to be included in control group 140 or treatment group 142.

[0051] The computing system 102 may determine the control version of the software application 122 used for the control-treatment experiment in any manner. In general, the control version of the software application 122 and the treatment version of the software application 122 may be the same version or newer than the version of the software application 122 installed at the computing device 110. In some examples, the computing system may determine the control version of the software application 122 as a version of the software application 122 that is fully released to the computing device (e.g., the version of the software application 122 that is most installed at the computing device 110) or the latest version of the software application 122 that precedes the new version of the software application 122 being tested. In some examples, the control version of the software application 122 may be a version created by a developer of the software application 122 that is most similar in size to the new version of the software application 122 or includes the same modules as the new version of the software application 122.

[0052] To test the new version of software application 122, the developer may deploy the new version of software application 122 at each of computing devices 110-2 in processing group 142. In a control-process experiment, the new version of software application 122 being tested is referred to as the process version of software application 122. The developer may use computing system 102 to send the process version of software application 122 to each of computing devices 110-2 in processing group 142, and each of computing devices 110-2 in processing group 142 may receive and install the process version of software application 122, such as by using application management module 116 to receive and install a software upgrade to upgrade software application 122 installed at computing device 110-2 to the process version of software application 122.

[0053] Computing system 102 may send the processed version of software application 122 to computing device 110 in any form. For example, computing system 102 may send a software update to computing device 110-2 in processing group 142 for updating software application 122 to the processed version of software application 122. The software update may include a package, an archive file, or any other technology for bundling code and resources, and may be compressed or uncompressed.

[0054] In some examples, a software update may include code and data for installing a processed version of a software application 122 in place of a currently installed version of the software application 122. That is, the application management module 116 may replace the currently installed version of the software application 122 with the processed version of the software application 122 in response to receiving the software update.

[0055] In some examples, the software update may include code and data for patching a currently installed version of the software application 122 to upgrade the software application 122 to a processed version of the software application 122. That is, the application management module 116 may patch the currently installed version of the software application 122 to update the currently installed version of the software application 122 to the processed version of the software application 122 in response to receiving the software update.

[0056] In some examples, the software update may include a modular update to the software application 122. Such a modular update may update specific features and functionality (referred to herein as modules) of the software application 122 to processed versions of such features and functionality. A software update for updating one or more modules of the software application 122 may include code and data for installing the processed version of the one or more modules in a currently installed version of the software application 122 and / or may include code and data for patching one or more modules in a currently installed version of the software application 122 to update those one or more modules in the software application 122 to the processed version of the one or more modules.

[0057] As part of testing the processed version of software application 122, computing system 102 may also send data to computing device 110-1 in control group 140. Sending data to computing device 110-1 in control group 140 may act as a placebo corresponding to the processed version of software application 122, and may enable developers to properly evaluate the performance of downloading and installing the processed version of the software application by computing devices in treatment group 142 relative to downloading and installing the data by computing devices in control group 140.

[0058] According to aspects of the present disclosure, during a control-treatment experiment for a software application 122, a testing module 104 of a computing system 102 may determine, for a computing device from computing device 110-1 in control group 140, a version of the software application 122 currently installed at the computing device. The testing module 104 may determine a version of the software application 122 to send to the computing device based on the currently installed version of the software application 122 at the computing device. Accordingly, the application distribution module 106 may send the determined version of the software application 122 to the computing device.

[0059] Sending the version of software application 122 to computing devices 110-1 in control group 140 may enable the developer of software application 122 to compare the performance of computing devices 110-2 in treatment group 142 in downloading and installing the treatment version of software application 122 with the performance of computing devices 110-1 in control group 140 in downloading and installing the version of software application 122. That is, during a control-treatment experiment, when computing system 102 receives a request to update software application 122 (such as an automatic update request or a manual update request) from a computing device in control group 140, in response, computing system 102 may send the version of software application 122 to the requesting computing device, and the developer may collect performance data regarding the download of software application 122.

[0060] In some examples, computing system 102 may send the processed version of software application 122 to computing devices 110-1 in control group 140 that already have the control version of software application 122 installed to trigger those computing devices to reinstall the control version of software application 122 from storage devices of the computing devices. That is, when computing system 102 determines that a computing device in control group 140 already has the control version of software application 122 installed, computing system 102 may determine that the control version of software application 122 should remain installed at the computing device.

[0061] Thus, computing system 102 may send a processed version of software application 122 to a computing device in control group 140 in response to determining that a control version of software application 122 is installed at the computing device, which causes the computing device to reinstall the control version of software application 122 from a storage device of the computing device. Triggering reinstallation of the control version of software application 122 may thus enable a developer of software application 122 to collect performance data regarding the installation of the control version of software application 122 at computing device 110-1 in control group 140 while enabling the control version of software application 122 to remain installed at the computing device.

[0062] Similarly, computing system 102 may send the control version of software application 122 to computing devices 110-1 in control group 140 that do not yet have the control version of software application 122 installed. Sending the control version of software application 122 to computing devices 110-1 in control group 140 that do not yet have the control version of software application 122 installed may trigger those computing devices to upgrade the installed version of software application 122 to the control version of software application 122. Thus, computing system 102 may send the control version of software application 122 to computing devices in control group 140 in response to determining that the version of software application 122 installed at the computing devices is not the control version of software application 122, which causes the computing devices to install the control version of software application 122 from storage devices of the computing devices.

[0063] In this manner, computing system 102 may be able to increase the number of computing devices 110-1 in control group 140 at which a control version of software application 122 is installed. Increasing the number of computing devices 110-1 in control group 140 at which a control version of software application 122 is installed may reduce any imbalance between the number of computing devices 110-1 in control group 140 at which a control version of software application 122 is installed and the number of computing devices 110-2 in processing group 142 at which a processing version of software application 122 is installed.

[0064] Computing system 102 may determine the version of software application 122 installed at computing device 110-1 in control group 140 in any suitable manner. For example, test data store 108 may store an indication of the version of software application 122 installed at each of computing devices 110, and computing system 102 may use test data store 108 to look up the version of software application 122 installed at computing device 110-1 in control group 140 as indicated by test data store 108.

[0065] The computing system 102 may send the determined version of the software application 122 to the computing device 110 in any form. As described above, the computing system 102 may send the determined version of the software application 122 in the form of a software update to update the software application 122 to the determined version of the software application 122. The software update may include a package, an archive file, or any other technology for bundling code and resources, and may be compressed or uncompressed.

[0066] In some examples, the software update may include code and data for installing the determined version of the software application 122 to replace the currently installed version of the software application 122. That is, the application management module 116 may replace the currently installed version of the software application 122 with the determined version of the software application 122 in response to receiving the software update.

[0067] In some examples, the software update may include code and data for patching a currently installed version of the software application 122 to upgrade the software application 122 to the determined version of the software application 122. That is, the application management module 116 may patch the currently installed version of the software application 122 to update the currently installed version of the software application 122 to the determined version of the software application 122 in response to receiving the software update.

[0068] In some examples, the software update may include a modular update to the software application 122. Such a modular update may update specific features and functionality (referred to herein as modules) of the software application 122 to determined versions of such features and functionality. A software update for updating one or more modules of the software application 122 may include code and data for installing the determined versions of the one or more modules in a currently installed version of the software application 122 and / or may include code and data for patching one or more modules in a currently installed version of the software application 122 to update those one or more modules in the software application 122 to the determined versions of the one or more modules.

[0069] exist Figure 1 In the example of FIG. 1 , computing system 102 may receive a request to update software application 122 installed at computing device 110-1A from computing device 110-1A in control group 140. In response, computing system 102 may determine the version of software application 122 installed at computing device 110-1A. If computing system 102 determines that the version of software application 122 installed at computing device 110-1A is a control version of software application 122, computing system 102 may send a software update for updating software application 122 to a processed version of software application 122 to computing device 110-1A to trigger computing device 110-1A to reinstall the control version of software application 122 from a storage device of computing device 110-1A. That is, computing device 110-1A may store data (such as an installation package, application package, etc.) for reinstalling a control version of software application 122 on a disk, and receipt of a processed version of software application 122 may enable computing device 110-1A to reinstall the control version of software application 122 from a storage device of computing device 110-1A.

[0070] Similarly, if computing system 102 determines that the version of software application 122 installed at computing device 110-1A is not the control version of software application 122, then in response, computing system 102 may send, to computing device 110-1A, a software update for updating software application 122 to the control version of software application 122. In response to receiving the software update for updating software application 122 to the control version of software application 122, computing device 110-1A may update software application 122 installed at computing device 110-1A to the control version of software application 122 using the received software update.

[0071] Computing device 110-1A may determine, in response to receiving the software update, whether to reinstall the currently installed version of software application 122 from a storage device of computing device 110-1A or upgrade to the received version of software application 122 based at least in part on determining whether the computing device is in a testing state (also referred to as a canary state). Computing device 110-1A may be in the testing state when a control-processing experiment is being performed to test the process version of software application 122, and may not be in the testing state when a control-processing experiment is not being performed to test the process version of software application 122. If computing device 110-1A determines that the computing device is not in the testing state, the computing device may install the version of software application 122 from computing system 102.

[0072] Computing device 110-1A may determine whether computing device 110-1A is in a test state via any suitable technique. In some examples, computing device 110-1A may receive an indication of whether computing device 110-1A is in a test state and a software update from computing system 102. In some examples, computing device 110-1A may store an indication of whether computing device 110-1A is in a test state, which computing device 110-1A may update each time computing device 110-1A enters or exits a test state. In some examples, computing device 110-1A may query computing system 102 whether computing device 110-1A is in a test state, and may receive a response from computing system 102 indicating whether computing device 110-1A is in a test state.

[0073] If computing device 110-1A determines that the computing device is in a testing state, computing device 110-1A may determine whether to reinstall the currently installed version of software application 122 from a storage device of computing device 110-1A or upgrade to the received version of software application 122 based at least in part on the version of software application 122 associated with the software update received from computing system 102. If computing device 110-1A determines that the software update received from computing system 102 is a software update for upgrading software application 122 to a processed version of software application 122, computing device 110-1A may determine to reinstall the currently installed version of software application 122 from a storage device of computing device 110-1A. In this manner, computing system 102 may trigger computing device 110-1A to reinstall the currently installed version of software application 122 from a storage device of computing device 110-1A by sending a software update for upgrading software application 122 to a processed version of software application 122 to computing device 110-1A.

[0074] In some examples, in addition to reinstalling the currently installed version of software application 122 from the storage of computing device 110-1A, computing device 110-1A may also install a copy of the processed version of software application 122 using the received update, and may store the copy of the processed version of software application 122 in the storage of computing device 110-1A. Installing and storing a copy of the processed version of software application 122 may enable computing device 110-1A to upgrade the version of software application 122 installed at computing device 110-1A to the processed version of software application 122 at a future time (such as when the processed version of software application 122 is released as the next official version of software application 122) without having to re-download the software update for upgrading software application 122 to the processed version of software application 122.

[0075] For example, when the processed version of software application 122 is released as the next official version of software application 122, computing system 102 may send an indication to computing device 110-1A that the processed version of software application 122 is released as the next official version of software application 122, the indication triggering computing device 110-1A to install the copy of the processed version of software application 122 stored at computing device 110-1A to replace the control version of software application 122 currently installed at computing device 110-1A. Such indication sent to computing device 110-1A may not necessarily include a software update for updating software application 122 to the processed version of software application 122, because a copy of the processed version of software application 122 may already be stored at computing device 110-1A. Bandwidth usage of computing device 110-1A is thereby reduced, because computing device 110-1A may not have to re-download a software update for updating software application 122 to the processed version of software application 122.

[0076] Similarly, if computing device 110-1A determines that the software update received from computing system 102 is a software update for upgrading software application 122 to a control version of software application 122, computing device 110-1A may use the software update to upgrade the currently installed version of software application 122 to the control version of software application 122. For example, computing device 110-1A may install the control version of software application 122 to replace the currently installed version of software application 122, update one or more modules of the currently installed version of software application 122 to process versions of the one or more modules, etc.

[0077] In some examples, regardless of the version of the software application 122 installed at the computing device, the computing system 102 can send a software update to the computing device, the software update including both code for updating the software application 122 to the process version of the software application 122 and code for updating the software application 122 to the control version of the software application 122. The computing devices in the control group 140 can use the code for updating the software application 122 to the control version of the software application 122 to update the software application 122 installed at the computing device to the control version of the software application. Similarly, the computing devices in the processing group 142 can use the code for updating the software application 122 to the process version of the software application 122 to update the software application 122 installed at the computing device to the process version of the software application 122.

[0078] Figure 2 is a block diagram showing additional details of an example computing device according to one or more aspects of the present disclosure. Figure 2 The computing device 210 in the embodiment is as follows Figure 1 The example of any one of the computing devices 110-1 in the control group 140 is described as follows

[0079] Figure 2 The computing device 210 may be an example of a mobile phone, a tablet computer, a laptop computer, a desktop computer, a server, a mainframe, a set-top box, a television, a wearable device, a home automation device or system, a gaming system, a media player, an e-book reader, a mobile television platform, a car navigation or infotainment system, or any other type of mobile, non-mobile, wearable, and non-wearable computing device configured to receive and output an indication of notification data. Figure 2 Only one particular example of computing device 210 is shown, and many other examples of computing device 210 may be used in other instances and may include a subset of the components included in the example computing device 210, or may include Figure 2 Additional components not shown.

[0080] like Figure 2 As shown in the example of , the computing device 210 includes a UIC 212, one or more processors 240, one or more input components 242, one or more communication units 244, one or more output components 246, and one or more storage components 248. The storage component 248 of the computing device 210 also includes an application management module 216, a software application 222, and a software application package 224.

[0081] Communication channel 250 can interconnect each of components 240, 212, 244, 246, 242, and 248 for inter-component communication (physically, communicatively, and / or operationally). In some examples, communication channel 250 can include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

[0082] One or more input components 242 of computing device 210 can receive input. Examples of input are tactile input, audio input, and video input. In one example, input components 242 of computing device 210 include a presence-sensitive display, a touch-sensitive screen, a mouse, a keyboard, a voice response system, a camera, a microphone, or any other type of device for detecting input from a person or a machine.

[0083] One or more output components 246 of computing device 210 can generate output. Examples of output are tactile output, audio output, and video output. In one example, output component 246 of computing device 210 includes a presence-sensitive display, a sound card, a video graphics adapter card, a speaker, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a light field display, a haptic motor, a linear actuator, or any other type of device for generating output to a person or machine.

[0084] The one or more communication units 244 of the computing device 210 can communicate with external devices via one or more wired and / or wireless networks by sending and / or receiving network signals over one or more networks. Examples of the one or more communication units 244 include a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and / or receive information. Other examples of the one or more communication units 244 can include a shortwave radio, a cellular data radio, a wireless network radio, and a universal serial bus (USB) controller.

[0085] The UIC 212 of the computing device 200 may be hardware used as an input and / or output device of the computing device 210. For example, the UIC 212 may include a display component, which may be a screen at which information is displayed by the UIC 212, and a presence-sensitive input component that may detect objects at and / or near the display component.

[0086] One or more processors 240 may implement functionality and / or execute instructions within the computing device 210. For example, one or more processors 240 on the computing device 210 may receive and execute instructions stored by the storage component 248 that perform the functionality of the modules 216 and 222 and the package 224. The instructions executed by the one or more processors 240 may cause the computing device 210 to store information within the storage component 248 during program execution. Examples of the one or more processors 240 include an application processor, a display controller, a sensor hub, and any other hardware configured to function as a processing unit. The one or more processors 240 may execute the instructions of the modules 216 and 222 and the package 224 to perform actions or functions. That is, the modules 216 and 222 and the package 224 may be operated by the one or more processors 240 to perform various actions or functions of the computing device 210.

[0087] One or more storage components 248 within computing device 210 may store information for processing during operation of computing device 210. That is, computing device 210 may store data accessed by modules 216 and 222 and package 224 during execution at computing device 210. In some examples, storage component 248 is temporary storage, meaning that the primary purpose of storage component 248 is not long-term storage. Storage component 248 on computing device 210 may be configured as volatile memory for short-term storage of information and therefore does not retain stored content in the event of a power outage. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

[0088] In some examples, storage component 248 also includes one or more computer-readable storage media. Storage component 248 can be configured to store larger amounts of information than volatile memory. Storage component 248 can also be configured as a non-volatile memory space for long-term storage of information and retaining information after power on / off cycles. Examples of non-volatile memory include a magnetic hard disk, an optical disk, a floppy disk, flash memory, or a form of electrically programmable memory (EPROM) or electrically erasable and programmable (EEPROM) memory. Storage component 248 can store program instructions and / or information (e.g., data) associated with modules 216 and 222 and package 224. Application management module 216 and software application 222 can be executed at one or more processors 240 to perform operations associated with Figure 1 The application management module 116 and the software application 122 have functions similar to those of the software application 122 .

[0089] The one or more processors 240 are configured to execute the application management module 216 to perform functions associated with receiving, managing, and otherwise handling software updates received at the computing device 210. Specifically, the application management module 216 can be executed at the one or more processors 240 to receive, manage, and otherwise handle software updates received at the computing device 210. Figure 1 The computing system 102 of the embodiment of the present invention receives a software update for updating the software application 222. Since the computing device 210 is in the control group for the control-treatment experiment, the application management module 216 can receive a software update for updating the software application 222 to the treatment version of the software application 222 or a software update for updating the software application 222 to the control version of the software application 222.

[0090] The application management module 216 may determine, in response to receiving the software update, whether to reinstall the currently installed version of the software application 222 or to upgrade to the received version of the software application 222. If the application management module 216 determines to reinstall the currently installed version of the software application 222, the application management module 216 may reinstall the currently installed version of the software application 222 using the software application package 224, which may be a software installation package for reinstalling the currently installed version of the software application 222 at the computing device 210.

[0091] In some examples, application management module 216 may determine whether to reinstall the currently installed version of software application 222 or upgrade to the received version of software application 222 from software application package 224 based at least in part on determining whether computing device 210 is in a testing state (also referred to as a canary state). Computing device 210 may be in a testing state when a control-processing experiment is being performed on software application 222, and may not be in a testing state when a control-processing experiment is not being performed on software application 222. If application management module 216 determines that the computing device is not in a testing state, the computing device may install the software application 222 that was downloaded from a computing system (e.g., Figure 1 The computing system 102) receives a version of the software application 222.

[0092] Application management module 216 may determine whether computing device 210 is in a test state via any suitable technique. In some examples, application management module 216 may receive an indication of whether computing device 210 is in a test state and a software update from a computing system. In some examples, application management module 216 may store an indication of whether computing device 210 is in a test state in storage component 248, which application management module 216 may update each time computing device 210 enters or exits a test state. In some examples, application management module 216 may query a computing system whether computing device 210 is in a test state, and may receive a response from the computing system indicating whether computing device 210 is in a test state.

[0093] If the application management module 216 determines that the computing device is in a testing state, the application management module 216 may determine whether to reinstall the currently installed version of the software application 222 from the software application package 224 stored in the storage component 248 or to upgrade the software application 222 using the received software update based at least in part on the version of the software application 222 associated with the software update received from the computing system. If the application management module 216 determines that the received software update is a software update for upgrading the software application 222 to a processed version of the software application 222, the application management module 216 may determine to reinstall the currently installed version of the software application 222 using the software application package 224 stored in the storage component 248. In this way, the computing system may trigger the computing device 210 to reinstall the currently installed version of the software application 222 using the software application package 224 stored in the storage component 248 by sending the software update for upgrading the software application 222 to the processed version of the software application 222 to the computing device 210.

[0094] In some examples, in addition to reinstalling the currently installed version of software application 222 using software application package 224 stored in storage component 248, computing device 110-1A may also install a copy of a processed version of software application 222 using the received update, and may store the copy of the processed version of software application 222 in storage component 248. Installing and storing a copy of the processed version of software application 222 may enable computing device 210 to upgrade the version of software application 222 installed at computing device 210 to the processed version of software application 222 at a future time without having to re-download the software update used to upgrade software application 222 to the processed version of software application 222.

[0095] Similarly, if the application management module 216 determines that the software update received from the computing system is a software update for upgrading the software application 222 to the control version of the software application 222, the application management module 216 may use the software update to upgrade the currently installed version of the software application 222 to the control version of the software application 222. For example, the application management module 216 may install the control version of the software application 222 to replace the currently installed version of the software application 222, update one or more modules of the currently installed version of the software application 222 to a process version of the one or more modules, and the like.

[0096] Figure 3 is a block diagram illustrating an example computing system according to one or more aspects of the present disclosure. Figure 3 The computing system 302 is as Figure 1 An example of a computing system 102 is described. Figure 3 Only one particular example of computing system 302 is shown, and many other examples of computing system 302 may be used in other instances and may include a subset of the components included in the example computing system 302, or may include Figure 3 For example, computing system 302 may include a server cluster, and each of the servers constituting the server cluster constituting computing system 302 may include the server cluster described herein. Figure 3 All or some of the components described in to perform the techniques disclosed herein.

[0097] like Figure 3 As shown in the example of FIG. 3 , computing system 302 includes one or more processors 340 , one or more communication units 342 , and one or more storage devices 348 . Storage device 348 includes test module 304 , application distribution module 306 , and test data storage area 308 .

[0098] The one or more processors 340 may implement functionality and / or execute instructions associated with the computing system 310. Examples of the one or more processors 340 include an application processor, a display controller, an auxiliary processor, one or more sensor hubs, and any other hardware configured to function as a processor, a processing unit, or a processing device. The test module 304 and the application distribution module 306 may be operated by the one or more processors 340 to perform various actions, operations, or functions of the computing system 310. For example, the one or more processors 340 of the computing system 310 may retrieve and execute instructions stored by the one or more storage components 348 that cause the one or more processors 340 to perform operations of the test module 304 and the application distribution module 306. The instructions, when executed by the one or more processors 340, may cause the computing system 310 to store information within the one or more storage components 348, for example, at the test data storage area 308.

[0099] The one or more communication units 342 of the computing system 310 can communicate with external devices via one or more wired and / or wireless networks by sending and / or receiving network signals over one or more networks. Examples of the one or more communication units 342 include a network interface card (e.g., such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a global positioning satellite (GPS) receiver, or any other type of device that can send and / or receive information. Other examples of the one or more communication units 342 can include a shortwave radio, a cellular data radio, a wireless network radio, and a universal serial bus (USB) controller.

[0100] Communication channel 350 can interconnect each of components 340, 342, and 348 for inter-component communication (physically, communicatively, and / or operationally). In some examples, communication channel 350 can include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

[0101] One or more storage components 348 within computing system 310 may store information for processing during operation of computing system 310 (e.g., computing system 310 may store data accessed by test module 304, application distribution module 306, and test data storage area 308 during execution at computing system 310). In some examples, one or more storage devices 348 are temporary storage, meaning that the primary purpose of one or more storage devices 348 is not long-term storage. In this example, one or more storage devices 348 may be configured as volatile memory for short-term storage of information, and therefore do not retain stored contents in the event of a power outage. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

[0102] In some examples, one or more storage devices 348 may also include one or more computer-readable storage media. In some examples, one or more storage devices 348 include one or more non-transitory computer-readable storage media. One or more storage devices 348 may be configured to store a larger amount of information than typically stored by volatile memory. One or more storage devices 348 may also be configured as non-volatile memory space for long-term storage of information and retain information after power on / off cycles. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or in the form of electrically programmable memory (EPROM) or electrically erasable and programmable (EEPROM) memory. One or more storage devices 348 may store program instructions and / or information (e.g., data) associated with test module 304, application distribution module 306, and test data storage area 308. Storage device 348 may include a memory configured to store data or other information associated with test module 304, application distribution module 306, and test data storage area 308.

[0103] The test data store 308 may be any data storage mechanism for storing structured data and / or unstructured data, and may be configured to store any data related to a control-process experiment on a software application. For example, the test data store 308 may store a version of the software application being tested, an indication of the version of the software application installed at a computing device, etc. Examples of the test data store 308 may include a database, a schema, a key-value store, etc.

[0104] The one or more processors 340 are configured to execute the application distribution module 306 and the test module 304 to perform a control-treatment experiment for the software application. Specifically, the one or more processors 340 are configured to execute the application distribution module 306 to distribute the software application to the computing devices via the one or more communication units 342. In some examples, the application distribution module 306 can generate a software update for updating the software application and send it to the computing device via the one or more communication units 342. That is, in some examples, when the application distribution module 306 (e.g., from the test module 304) receives a request to send a version of the software application to one or more computing devices, the application distribution module 306 can generate a software update for updating the software application to the version of the software application, and can send the software update to the one or more computing devices.

[0105] The one or more processors 340 are configured to execute the test module 304 to perform a control-processing experiment for the software application. The processing module 304 may receive a computing device (e.g., Figure 1 10) at the processing group, and in response, application distribution module 306 can use application distribution module 306 to send the processed version of the software application to the computing devices in the processing group via one or more communication units 342. For example, application distribution module 306 can generate a software update for updating the software application to the processed version of the software application in response to receiving a request from testing module 304 to send the processed version of the software application, and can send the software update for updating the software application to the processed version of the software application to the computing devices in the processing group.

[0106] Correspondingly, when the test module 304 uses the application distribution module 306 to send the processing version of the software application to the computing devices in the processing group, the test module 304 may also use the application distribution module 306 to send one or more versions of the software application to the computing devices in the control group for testing of the software application. Specifically, the test module 304 may determine the version of the software application installed at the computing device for each computing device in the control group for testing of the software application. If the test module 304 determines that the version of the software application installed at the computing device in the control group is not the control version of the software application, the test module 304 may send the control version of the software application to the computing device via the application distribution module 306.

[0107] Similarly, if the testing module 304 determines that the version of the software application installed at the computing device in the control group is the control version of the software application, the testing module 304 may send the processed version of the software application to the computing device via the application distribution module 306. Sending the processed version of the software application to the computing device in the control group that has the control version of the software application installed may trigger the computing device to reinstall the control version of the software application from the storage device of the computing device.

[0108] As described above, the application distribution module 306 can send the version of the software application (e.g., a processed version of the software application or a controlled version of the software application) to one or more computing devices in the control group via the one or more communication units 342 in any form. The processed version of the software application is sent to the computing devices in the processing group via the one or more communication units 342. For example, the application distribution module 306 can generate a software update for updating the software application to the specified version of the software application in response to receiving a request from the testing module 304 to send the specified version of the software application to the computing device, and can send the software update for updating the software application to the specified version of the software application to the computing device.

[0109] FIG. 4A to FIG. 4B is a block diagram illustrating software testing according to various aspects of the present disclosure. Figure 4A As shown, Figure 1 The computing system 402 of the example computing system 102 may be used as Figure 1 Computing devices 410A and 410B of the example of computing device 110 of FIG. 1 communicate to perform testing of software application 422. Computing device 410A may be in a control group for performing a control-treatment experiment on software application 422, while computing device 410B may be in a treatment group for performing a control-release experiment.

[0110] The software application 422 may include modules A, B, C, and D, and the control versions of the software application 422 may include version 2 of module A, version 2 of module B, version 1 of module C, and version 1 of module D. The computing devices 410A and 410B may each be installed with a control version of the software application 422. Therefore, each of the software applications 422 installed at the computing devices 410A and 410B includes version 2 of module A, version 2 of module B, version 1 of module C, and version 1 of module D.

[0111] As part of performing the control-process experiment, computing system 402 may deploy a process version of software application 422 to computing devices in a process group for the control-process experiment. While the control version of software application 422 includes version 2 of module A, version 2 of module B, version 1 of module C, and version 1 of module D, the process version of software application 422 may include version 3 of module A, version 2 of module B, version 3 of module C, and version 1 of module D. Thus, upgrading the control version of software application 422 to the process version of software application 422 may include upgrading version 2 of module A in the control version of software application 422 to version 3 of module A, and upgrading version 1 of module C in the control version to version 3 of module C.

[0112] Thus, computing system 402 may deploy the processed version of software application 422 in the form of a software update that updates version 2 of module A in the control version of software application 422 to version 3 of module A, and updates version 1 of module C in the control version to version 3 of module C. In some examples, the software update may include code for installing the complete processed version of software application 422. That is, the software update may install each of version 3 of module A, version 2 of module B, version 3 of module C, and version 1 of module D included in the processed version of software application 422.

[0113] In some examples, the software update may update the control version of the software application 422 to the processed version of the software application 422 by including code for updating less than the entire control version of the software application 422 to the processed version of the software application 422. That is, the software update may instead update only the portion of the control version of the software application 422 that is different from the processed version of the software application 422. Figure 4A In the example, since only modules A and C of the control version of software application 422 are different from the processed version of software application 422, the software update can update modules A and C of the control version of software application 422 to the processed versions of modules A and C.

[0114] Thus, computing system 402 may send to computing device 410B a software update 450B for updating the control versions of modules A and C of software application 422 to processed versions of modules A and C (e.g., version 3 of module A and version 3 of module C). Software update 450B may include code for installing the processed versions of modules A and C, such as by replacing the control versions of modules A and C with the processed versions of modules A and C, patching the control versions of modules A and C to the processed versions of modules A and C, or otherwise updating the control versions of modules A and C to the processed versions of modules A and C.

[0115] Computing device 410B may, in response to receiving software update 450B, apply software update 450B to update the control version of software application 422 to the processed version of software application 422. That is, computing device 410B may replace the control versions of modules A and C with the processed versions of modules A and C, patch the control versions of modules A and C to the processed versions of modules A and C, or otherwise update the control versions of modules A and C to the processed versions of modules A and C, resulting in the installation of a processed version of software application 422 at computing device 410B that includes version 3 of module A, version 2 of module B, version 3 of module C, and version 1 of module D.

[0116] Computing system 402 may correspondingly send software update 450A to computing device 410A in the control group for the control-treatment experiment. Computing system 402 may determine that the control version of software application 422 is installed at computing device 410A, and in response, may send software update 450A to computing device 410A for updating software application 422 to the treatment version of software application 422 to trigger computing device 410A to reinstall the control version of software application 422 from the storage device of computing device 410A.

[0117] exist Figure 4A In the example of , software update 450A sent to computing device 410A includes code for updating the same modules as software update 450B sent to computing device 410B. That is, software update 450A includes code for installing processed versions of modules A and C, such as by replacing controlled versions of modules A and C with processed versions of modules A and C, patching controlled versions of modules A and C to processed versions of modules A and C, or otherwise updating controlled versions of modules A and C to processed versions of modules A and C.

[0118] Computing device 410A may determine, in response to receiving software update 450A, whether to install software update 450A or to reinstall the control version of software application 422 from a storage device of computing device 410A. For example, computing device 410A may determine whether a currently installed version of software application 422 is a control version of software application 422, and if the currently installed version of software application 422 is the control version of software application 422, may determine to reinstall the current version of software application 422.

[0119] If computing device 410A determines to reinstall the control version of software application 422, computing device 410A may reinstall the control version of software application 422 from a storage device of computing device 410A. That is, computing device 410A may store data (such as an installation package, an application package, etc.) for reinstalling the control version of software application 422 on a disk. Thus, computing device 410A may reinstall the control version of software application 422 using the data for reinstalling the control version of software application 422 stored in a disk storage device (e.g., a hard drive) of computing device 410A in response to receiving software update 450A. In some examples, computing device 410A may also store software update 450A in a storage device of computing device 410A, which may be used by computing device 410A at a later time to upgrade the control version of software application 422 to a processed version of software application 422 without having to re-download the software update for upgrading software application 422 to the processed version of software application 422.

[0120] In some examples, when computing system 402 determines to send a software update for updating software application 422 to a control version of software application 422, computing system 402 may generate a software update that includes code for updating the same modules updated by the software update for updating software application 422 to a process version of software application 422. That is, even if the software update for updating software application 422 to a control version of software application 422 updates some but not all of the modules updated by the software update for updating software application 422 to a process version of software application 422, the software update for updating software application 422 to a control version of software application 422 may still include code for updating the same modules updated by the software update for updating software application 422 to a process version of software application 422. By including code for updating the same module that is updated by the software update for updating software application 422 to the processed version of software application 422, the software update for updating software application 422 to the controlled version of software application 422 can be closer in size to the software update for updating software application 422 to the processed version of software application 422, thereby reducing the amount of skew introduced by the size difference between the software updates when comparing performance data associated with the downloads of these software updates.

[0121] like Figure 4B As shown, Figure 1 The computing system 402 of the example computing system 102 may be used as Figure 1Computing devices 410C and 410D of the example of computing device 110 communicate to perform testing of software application 422. Computing device 410C may be in a control group for performing a control-treatment experiment on software application 422, while computing device 410D may be in a treatment group for performing a control-release experiment.

[0122] Software application 422 may include modules A, B, C, and D, and the control versions of software application 422 may include version 2 of module A, version 2 of module B, version 1 of module C, and version 1 of module D. However, neither computing devices 410C nor 410D may have installed the control version of software application 422. Instead, each of computing devices 410C and 410B may have installed an older version of software application 422 including version 1 of module A, version 1 of module B, version 1 of module C, and version 1 of module D.

[0123] As part of performing the control-treatment experiment, computing system 402 may deploy a treatment version of software application 422 to computing devices in a treatment group for the control-treatment experiment. While the versions of software application 422 installed at computing device 410D include version 1 of module A, version 1 of module B, version 1 of module C, and version 1 of module D, the treatment version of software application 422 may include version 3 of module A, version 2 of module B, version 3 of module C, and version 1 of module D. Thus, upgrading the control version of software application 422 to the treatment version of software application 422 may include upgrading version 1 of module A in the control version of software application 422 to version 3 of module A, upgrading version 1 of module B to version 2 of module B, and upgrading version 1 of module C in the control version to version 3 of module C.

[0124] Thus, computing system 402 may deploy a processed version of software application 422 to computing device 410D in the form of software update 450D, which updates version 1 of module A in the control version of software application 422 to version 3 of module A, updates version 1 of module B to version 2 of module B, and updates version 1 of module C in the control version to version 3 of module C. In some examples, the software update may include code for installing the complete processed version of software application 422. That is, the software update may install each of version 3 of module A, version 2 of module B, version 3 of module C, and version 1 of module D included in the processed version of software application 422. Computing device 410B may apply software update 450D in response to receiving software update 450D to update the control version of software application 422 installed at computing device 410B to the processed version of software application 422.

[0125] Computing system 402 may correspondingly send software update 450C to computing device 410C in the control group for the control-treatment experiment. Computing system 402 may determine that the version of software application 422 installed at computing device 410C is not the control version of software application 422, and in response, may send software update 450C to computing device 410C for updating software application 422 to the control version of software application 422.

[0126] The control version of software application 422 includes version 2 of module A, version 2 of module B, version 1 of module C, and version 1 of module D. Therefore, a software update for updating software application 422 to the control version of software application 422 may include at least code for updating version 1 of module A to version 2 of module A and for updating version 1 of module B to version 2 of module B in the control version of software application 422.

[0127] However, unlike update 450D, a software update that includes only code for updating version 1 of module A in software application 422 to version 2 of module A and for updating version 1 of module B in software application 422 to version 2 of module B does not include code for updating module C. Therefore, computing system 402 may send software update 450C to computing device 410C, the software update including code for updating version 1 of module A in software application 422 to version 2 of module A and for updating version 1 of module B in software application 422 to version 2 of module B, and code for updating module C in software application 422 to version 3 of module C.

[0128] Computing device 410C may determine, in response to receiving software update 450C, whether to install software update 450C or to reinstall the currently installed version of software application 422 from a storage device of computing device 410C. For example, computing device 410C may determine whether the currently installed version of software application 422 is a control version of software application 422, and if the currently installed version of software application 422 is a control version of software application 422, may determine to reinstall the current version of software application 422.

[0129] exist Figure 4BIn the example of , computing device 410C may determine that the currently installed version of software application 422 is not the control version of software application 422, and therefore may install software update 450C to update the version of software application 422 installed at computing device 410C to the control version of software application 422. Specifically, computing device 410C may utilize software update 450C to update version 1 of module A in software application 422 to version 2 of module A and to update version 1 of module B to version 2 of module B. Additionally, since the control version of software application 422 does not include updated versions of modules C and D, computing device 410C may also reinstall version 1 of module C and version 1 of module D from a storage device of computing device 410C as part of updating software application 422 to the control version of software application 422. In this way, computing device 410C may update an old version of software application 422 to the control version of software application 422.

[0130] Figure 5 is a flowchart illustrating example operations performed by an example computing system configured to perform software testing according to one or more aspects of the present disclosure. Figure 1 Environment 100 and Figure 3 The context of computing system 302 is described in Figure 5 .

[0131] like Figure 5 As shown, one or more processors 340 of computing system 302 may determine that software application 122 installed at computing device 110-1A in control group 140 associated with a control-treatment experiment for software application 122 is a control version of software application 122 (502). One or more processors 340 may determine a software update to be sent to computing device 110-1A to update software application 122 to a treatment version of software application 122 based at least in part on computing device 110-1A being in control group 140 and software application 122 installed at computing device 110-1A being a control version of software application 122 (504). One or more processors 340 may send the software update to computing device 110-1A to trigger computing device 110-1A to reinstall the control version of software application 122 at computing device 110-1A from a storage device of computing device 110-1A in response to receiving the software update (506).

[0132] Figure 6 is a flow chart illustrating example operations performed by an example computing device in a control group of a control-processing experiment for testing a software application in accordance with one or more aspects of the present disclosure. Figure 1 Environment 100 and Figure 2 The context of the computing device 210 describes Figure 6 .

[0133] like Figure 6 As shown, one or more processors 240 of computing device 210 may receive a software update for updating software application 222 from computing system 102, wherein computing device 210 is in a control group associated with a control-treatment experiment for software application 222, and wherein a control version of software application 222 is installed at computing device 210 (602). One or more processors 240 may determine that the software update is for updating software application 222 to a treatment version of software application 222 (604). One or more processors 240 may reinstall the control version of software application 222 from storage device 248 of computing device 210 at computing device 210 in response to determining that the software update is for updating software application 222 to a treatment version of software application 222 (606).

[0134] Aspects of the present disclosure include the following examples.

[0135] Example 1: A method, comprising: determining, by one or more processors of a computing system, that a software application installed at a computing device in a control group associated with a control-treatment experiment for a software application is a control version of the software application; determining, by the one or more processors, a software update to be sent to the computing device for updating the software application to a treatment version of the software application based at least in part on the fact that the computing device is in the control group and that the software application installed at the computing device is the control version of the software application; and sending, by the one or more processors, the software update to the computing device to trigger the computing device to reinstall the control version of the software application at the computing device from a storage device of the computing device in response to receiving the software update.

[0136] Example 2: The method as described in Example 1 also includes sending the software update for updating the software application to the processed version of the software application by the one or more processors to multiple computing devices in a processing group associated with the control-processing experiment for the software application.

[0137] Example 3: A method as described in any of Examples 1 and 2, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the processed version of the software application.

[0138] Example 4: The method as described in any of Examples 1 to 3, further includes: determining, by the one or more processors, whether the software application installed at a second computing device in the control group is the control version of the software application; and in response to determining that the software application installed at the second computing device is not the control version of the software application, sending, by the one or more processors, to the second computing device a second software update for updating the software application to the control version of the software application.

[0139] Example 5: The method as described in any of Examples 1 to 4 further includes: determining, by the one or more processors, the control version of the software application as one of the following: a version of the software application that is fully released to a computing device or a latest version of the software application before the processed version of the software application is deployed to the multiple computing devices in the processing group.

[0140] Example 6: A method as described in any of Examples 1 to 5, wherein sending the software update also includes: receiving, by the one or more processors, an automatic update request associated with the software application from the computing device; and in response to receiving the automatic update request from the computing device, sending the software update to the computing device by the one or more processors to trigger the computing device to reinstall the control version of the software application from the storage device of the computing device at the computing device in response to receiving the software update.

[0141] Example 7: A method as described in any of Examples 1 to 6, wherein sending the software update to the computing device also triggers the computing device to store a copy of the processed version of the software application in the storage device of the computing device, and the method also includes sending an indication by the one or more processors to the computing device that the processed version of the software application is to be released as the next official version of the software application, the indication causing the computing device to install the copy of the processed version of the software application from the computing device to replace the control version of the software application.

[0142] Example 8: A computing system comprising: a memory storing instructions; and one or more processors executing the instructions to: determine that a software application installed at a computing device in a control group associated with a control-treatment experiment for a software application is a control version of the software application; determine a software update to be sent to the computing device to update the software application to a treatment version of the software application based at least in part on the fact that the computing device is in the control group and the software application installed at the computing device is the control version of the software application; and send the software update to the computing device to trigger the computing device to reinstall the control version of the software application at the computing device from a storage device of the computing device in response to receiving the software update.

[0143] Example 9: A computing system as described in Example 8, wherein the one or more processors also execute the instructions to: send the software update for updating the software application to a processed version of the software application to multiple computing devices in a processing group associated with the control-processing experiment for the software application.

[0144] Example 10: A computing system as described in any of Examples 8 and 9, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the processed version of the software application.

[0145] Example 11: A computing system as described in any of Examples 8 to 10, wherein the one or more processors further execute the instructions to: determine that the software application installed at a second computing device in the control group is the control version of the software application; and in response to determining that the software application installed at the second computing device is not the control version of the software application, send a second software update to the second computing device for updating the software application to the control version of the software application.

[0146] Example 12: A computing system as described in any of Examples 8 to 11, wherein the one or more processors also execute the instructions to: determine the control version of the software application to be one of the following: a version of the software application that is fully released to a computing device or a latest version of the software application before the processed version of the software application is deployed to the multiple computing devices in the processing group.

[0147] Example 13: A computing system as described in any of Examples 8 to 12, wherein in order to send the software update, the one or more processors also execute the instructions to: receive an automatic update request associated with the software application from the computing device; and in response to receiving the automatic update request from the computing device, send the software update to the computing device to trigger the computing device to reinstall the control version of the software application from the storage device of the computing device at the computing device in response to receiving the software update.

[0148] Example 14: A computing system as described in any of Examples 8 to 13, wherein the one or more processors also execute the instructions to send the software update to the computing device to further trigger the computing device to store a copy of the processed version of the software application in the storage device of the computing device, and also includes sending an instruction by the one or more processors to the computing device that the processed version of the software application is released as the next official version of the software application, the instruction causing the computing device to install the copy of the processed version of the software application from the computing device to replace the control version of the software application.

[0149] Example 15: A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a computing system, cause the one or more processors to: determine that a software application installed at a computing device in a control group associated with a control-treatment experiment for a software application is a control version of the software application; determine a software update to be sent to the computing device to update the software application to a treatment version of the software application based at least in part on the computing device being in the control group and the software application installed at the computing device being the control version of the software application; and send the software update to the computing device to trigger the computing device to reinstall the control version of the software application at the computing device from a storage device of the computing device in response to receiving the software update.

[0150] Example 16: The non-transitory computer-readable storage medium of Example 15, wherein the instructions further cause the one or more processors to: send the software update for updating the software application to the processed version of the software application to a plurality of computing devices in a processing group associated with the control-processing experiment for the software application.

[0151] Example 17: A non-transitory computer-readable storage medium as described in any of Examples 15 and 16, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the processed version of the software application.

[0152] Example 18: A non-transitory computer-readable storage medium as described in any of Examples 15 to 18, wherein the instructions further cause the one or more processors to: determine that the software application installed at a second computing device in the control group is not the control version of the software application; and in response to determining that the second computing device is in the control group and in response to the software application installed at the second computing device being not the control version of the software application, send a second software update to the second computing device for updating the software application to the control version of the software application.

[0153] Example 19: A non-transitory computer-readable storage medium as described in any of Examples 15 to 18, wherein the instructions further cause the one or more processors to: determine the control version of the software application to be one of: a version of the software application that is fully released to a computing device or a latest version of the software application before the processed version of the software application is deployed to the multiple computing devices in the processing group.

[0154] Example 20: A non-transitory computer-readable storage medium as described in any of Examples 15 to 19, wherein the instructions that cause the one or more processors to send the software update also cause the one or more processors to: receive an automatic update request associated with the software application from the computing device; and in response to receiving the automatic update request from the computing device, send the software update to the computing device to trigger the computing device to reinstall the control version of the software application from the storage device of the computing device at the computing device in response to receiving the software update.

[0155] Example 21: A method, comprising: receiving, by one or more processors of a computing device, a software update for updating a software application from a computing system, wherein the computing device is in a control group associated with a control-treatment experiment for the software application, and wherein a control version of the software application is installed at the computing device; determining, by the one or more processors, whether the software update is for updating the software application to a treatment version of the software application; and in response to determining that the software update is for updating the software application to the treatment version of the software application, reinstalling, by the one or more processors, the control version of the software application from a storage device of the computing device at the computing device.

[0156] Example 22: A method as described in Example 21, wherein a software installation package for installing the control version of the software application is stored at the storage device of the computing device, and wherein reinstalling the control version of the software application also includes: using the software installation package by the one or more processors to reinstall the control version of the software application at the computing device.

[0157] Example 23: A method as described in any of Examples 21 and 22, wherein reinstalling the control version of the software application also includes: using the software update by the one or more processors to update the copy of the control version of the software application to the processed version of the software application; and storing the copy of the processed version of the software application in the storage device of the computing device by the one or more processors.

[0158] Example 24: A method as described in Example 23, wherein reinstalling the control version of the software application at the computing device further includes: in response to determining that the computing device is in a testing state, reinstalling the control version of the software application from the storage device of the computing device at the computing device by the one or more processors.

[0159] Example 25: The method of Example 24 further comprises receiving, by the one or more processors from the computing system, an indication that the computing device is in the testing state.

[0160] Example 26: A method as described in Example 25, wherein receiving the indication that the computing device is in the test state also includes: sending a query by the one or more processors to the computing system regarding whether the computing device is in the test state; and in response to sending the query, receiving the indication that the computing device is in the test state from the computing system by the one or more processors.

[0161] Example 27: The method as described in any of Examples 24 to 26 further includes: determining, by the one or more processors, whether the computing device is in the testing state; and in response to determining that the computing device is no longer in the testing state, installing, by the one or more processors, the copy of the processed version of the software application stored at the computing device to replace the control version of the software application.

[0162] Example 28: A method as described in any of Examples 21 to 27, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the processed version of the software application.

[0163] Example 29: The method of any one of Examples 21 to 28, wherein receiving the software update is in response to sending, by the one or more processors, an automatic update request for the software application to the computing system.

[0164] Example 30: A computing device comprising: a memory storing instructions; a storage device; and one or more processors executing the instructions to: receive a software update for updating a software application from a computing system, wherein the computing device is in a control group associated with a control-treatment experiment for the software application, and wherein a control version of the software application is installed at the computing device; determine whether the software update is for updating the software application to a treatment version of the software application; and in response to determining that the software update is for updating the software application to the treatment version of the software application, reinstall the control version of the software application from the storage device of the computing device at the computing device.

[0165] Example 31: A computing device as described in Example 30, wherein the storage device of the computing device is configured to store a software installation package for installing the control version of the software application, and wherein in order to reinstall the control version of the software application, the one or more processors are also configured to: use the software installation package to reinstall the control version of the software application at the computing device.

[0166] Example 32: A computing device as described in any of Examples 30 and 31, wherein in order to reinstall the control version of the software application, the one or more processors also execute the instructions to: use the software update to update the copy of the control version of the software application to the processed version of the software application; and store the copy of the processed version of the software application in the storage device of the computing device.

[0167] Example 33: A computing device as described in any of Example 32, wherein reinstalling the control version of the software application at the computing device further includes: in response to determining that the computing device is in a testing state, reinstalling the control version of the software application at the computing device from the storage device of the computing device.

[0168] Example 34: The computing device of Example 33, wherein the one or more processors are further configured to: receive an indication from the computing system that the computing device is in the testing state.

[0169] Example 35: A computing device as described in Example 34, wherein, in order to receive the indication that the computing device is in the test state, the one or more processors are also configured to: send a query to the computing system regarding whether the computing device is in the test state; and in response to sending the query, receive the indication that the computing device is in the test state from the computing system.

[0170] Example 36: A computing device as described in any of Examples 33 to 35, wherein the one or more processors are further configured to: determine whether the computing device is in the testing state; and in response to determining that the computing device is no longer in the testing state, reinstall the copy of the processed version of the software application stored at the computing device to replace the control version of the software application.

[0171] Example 37: A computing device as described in any of Examples 30 to 36, wherein the software update for updating the software application includes code for updating the software application to the control version of the software application and code for updating the software application to the processed version of the software application.

[0172] Example 38: A computing device as described in any of Examples 30 to 37, wherein the one or more processors are configured to receive the software update in response to sending an automatic update request for the software application to the computing system.

[0173] Example 39: A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a computing device, cause the one or more processors to: receive a software update for updating a software application from a computing system, wherein the computing device is in a control group associated with a control-treatment experiment for the software application, and wherein a control version of the software application is installed at the computing device; determine whether the software update is for updating the software application to a treatment version of the software application; and in response to determining that the software update is for updating the software application to the treatment version of the software application, reinstall the control version of the software application from a storage device of the computing device at the computing device.

[0174] Example 40: The non-transitory computer-readable storage medium of Example 39, wherein the instructions that cause the one or more processors to reinstall the control version of the software application also cause the one or more processors to: use the software update to update the copy of the control version of the software application to the processed version of the software application; and store the copy of the processed version of the software application in the storage device of the computing device.

[0175] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, disk storage devices or other magnetic storage devices, flash memory or any other storage medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to send instructions from a website, server or other remote source, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave) is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but are directed to non-temporary tangible storage media. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0176] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules. Moreover, the techniques may be fully implemented in one or more circuits or logic elements.

[0177] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but the various components, modules, or units are not necessarily required to be implemented by different hardware units. Rather, as described above, the various units may be combined in a hardware unit or provided by a collection of interoperable hardware units including one or more processors as described above in combination with appropriate software and / or firmware.

[0178] Various embodiments have been described. These and other embodiments are within the scope of the following claims.

Claims

1. A method, include: determining, by one or more processors of a computing system, that the software application installed at a computing device in a control group associated with a control-treatment experiment for the software application is a control version of the software application; determining, by the one or more processors, a software update to be sent to the computing device for updating the software application to a process version of the software application based at least in part on the computing device being in the control group and the software application installed at the computing device being the control version of the software application; as well as The software update is sent, by the one or more processors, to the computing device to trigger the computing device to reinstall the control version of the software application at the computing device from a storage device of the computing device in response to receiving the software update.

2. The method according to claim 1, further comprising: include: The software update for updating the software application to the process version of the software application is sent by the one or more processors to a plurality of computing devices in a process group associated with the control-process experiment for the software application.

3. The method of any one of claims 1 and 2, wherein the software update for updating the software application comprises code for updating the software application to the control version of the software application and code for updating the software application to the processed version of the software application.

4. The method according to any one of claims 1 to 3, further comprising: include: determining, by the one or more processors, whether the software application installed at a second computing device in the control group is the control version of the software application; as well as In response to determining that the software application installed at the second computing device is not the control version of the software application, a second software update for updating the software application to the control version of the software application is sent by the one or more processors to the second computing device.

5. The method according to any one of claims 1 to 4, further comprising: include: The control version of the software application is determined, by the one or more processors, to be one of: a fully released version of the software application to a computing device or a latest version of the software application that precedes the processed version of the software application.

6. The method of any one of claims 1 to 5, wherein sending the software update further comprises: include: receiving, by the one or more processors, from the computing device, an automatic update request associated with the software application; as well as In response to receiving the automatic update request from the computing device, the one or more processors send the software update to the computing device to trigger the computing device to reinstall the control version of the software application from the storage device of the computing device at the computing device in response to receiving the software update.

7. The method of any one of claims 1 to 6, wherein sending the software update to the computing device also triggers the computing device to store a copy of the processed version of the software application in the storage device of the computing device, the method further include: An indication is sent by the one or more processors to the computing device that the processed version of the software application is to be released as the next official version of the software application, the indication causing the computing device to install the copy of the processed version of the software application stored at the computing device to replace the control version of the software application.

8. A computing system, include: a memory storing instructions; as well as One or more processors, the one or more processors executing the instructions to perform the method according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium comprising instructions which, when executed by one or more processors of a computing system, cause the one or more processors to perform the method of any one of claims 1 to 7.

10. A method, include: receiving, by one or more processors of a computing device from a computing system, a software update for updating a software application, wherein the computing device is in a control group associated with a control-treatment experiment for the software application, and wherein a control version of the software application is installed at the computing device; determining, by the one or more processors, whether the software update is for updating the software application to a processed version of the software application; as well as In response to determining that the software update is for updating the software application to the processed version of the software application, reinstalling, by the one or more processors, the processed version of the software application from a storage device of the computing device at the computing device.

11. The method of claim 10, wherein a software installation package for installing the control version of the software application is stored at the storage device of the computing device, and wherein reinstalling the control version of the software application further comprises: include: The software installation package is used by the one or more processors to reinstall the control version of the software application at the computing device.

12. The method of any one of claims 10 and 11, wherein reinstalling the control version of the software application further comprises: include: using the software update, by the one or more processors, to update a copy of the control version of the software application to the processed version of the software application; as well as A copy of the processed version of the software application is stored, by the one or more processors, in the storage device of the computing device.

13. The method of claim 12, wherein reinstalling the control version of the software application at the computing device further comprises: include: In response to determining that the computing device is in the testing state, the control version of the software application is reinstalled at the computing device from the storage device of the computing device by the one or more processors.

14. The method of claim 13, further comprising: include: An indication that the computing device is in the testing state is received by the one or more processors from the computing system.

15. The method of claim 14, wherein receiving the indication that the computing device is in the testing state further comprises: include: sending, by the one or more processors, a query to the computing system regarding whether the computing device is in the testing state; And in response to sending the query, receiving, by the one or more processors from the computing system, the indication that the computing device is in the testing state.

16. The method according to any one of claims 13 to 15, further comprising: include: determining, by the one or more processors, whether the computing device is in the testing state; as well as In response to determining that the computing device is no longer in the testing state, the copy of the processed version of the software application stored at the computing device is installed by the one or more processors to replace the control version of the software application.

17. The method of any one of claims 10 to 16, wherein the software update for updating the software application comprises code for updating the software application to the control version of the software application and code for updating the software application to the process version of the software application.

18. The method of any one of claims 10 to 17, wherein receiving the software update is in response to sending, by the one or more processors, an automatic update request for the software application to the computing system.

19. A computing system, include: a memory storing instructions; as well as One or more processors, the one or more processors executing the instructions to perform the method of any one of claims 10 to 18.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a computing device, cause the one or more processors to perform the method of any one of claims 10 to 18.

Images