Application Release Method, Device, Electronic Device and Medium Based on Bucketing Experiment

Abstract

The present application discloses an application release method, device, electronic device and medium based on a bucketing experiment. The method includes: constructing a test sub-experiment for the version to be released this time in the test mother experiment of the application, and the test sub-experiment includes multiple buckets; obtaining a first solidification result, which is obtained by performing a bucketing test experiment on the test sub-experiment of the previous version to be released; using the first solidification result as the initial value of the test sub-experiment of the version to be released this time, performing a bucketing test experiment on the test sub-experiment of the version to be released this time, and obtaining a second solidification result; updating the application according to the first solidification result and the second solidification result. By establishing interrelated mother and son test experiments, the present application enables the solidified content to still enjoy the gray-scale release ability, realizes the version replacement without having to re-establish the overall test experiment, and greatly reduces the storage pressure and transmission pressure on the backend, saving a large amount of backend resources.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to an application release method, apparatus, electronic device, and medium based on bucket experiments. Background Art

[0002] Bucket experiments are used to create two (A / B) or multiple (A / B / N) versions of a product application, page, or process. In the same time dimension, visitor groups with the same or similar compositions are divided from the entire user traffic to randomly access these different experimental versions. Different experimental schemes are set for different experimental versions, and user experience data and result data of each group are collected. By observing and analyzing the effects of experimental metrics, product iteration is promoted through data-driven means, algorithm effects are verified, output values are obtained, etc., to draw experimental conclusions. Finally, the version with better effects is analyzed and evaluated and officially adopted. This experimental method is widely used in Internet products for product iteration and optimization.

[0003] However, in the prior art, once the functions of a released application are fully solidified, when there are again change requirements, the functions cannot be changed, and only the application needs to be re-released. This method has many drawbacks. For example, during the process of re-releasing the application, code needs to be rewritten, consuming a large amount of human and time costs; each update needs to be sent to the terminal, consuming backend storage resources; and a large amount of experimental data needs to be carried by the terminal interface each time, significantly increasing the interface transmission time. Summary of the Invention

[0004] In view of the above situation, embodiments of this application propose an application release method, apparatus, electronic device, and medium based on bucket experiments. By establishing interrelated test parent-child experiments, the solidified content can still enjoy the gray-scale release ability, and version replacement can be achieved without the need to re-establish the overall test experiment.

[0005] In a first aspect, embodiments of this application provide an application release method based on bucket experiments, including:

[0006] Construct a test sub-experiment of the version to be released this time in the test parent-experiment of the application, where the test sub-experiment includes multiple buckets, and each bucket corresponds to a sub-version of the version to be released this time;

[0007] Obtain a first solidification result, where the first solidification result is obtained through a bucket test experiment on the test sub-experiment of the previous version to be released;

[0008] Use the first solidification result as a bucket in the test sub-experiment of the version to be released this time, and use this bucket as the initial value of the test sub-experiment of the version to be released this time to conduct a bucket test experiment on the test sub-experiment of the version to be released this time to obtain a second solidification result;

[0009] Update the application according to the first solidification result and the second solidification result. In a second aspect, an embodiment of the present application further provides an application release device based on a bucket experiment. The device includes:

[0010] A construction unit for constructing a test sub-experiment of the version to be released this time in the test mother experiment of the application, where the test sub-experiment includes multiple buckets, and each bucket corresponds to a sub-version of the version to be released this time;

[0011] An acquisition unit for acquiring a first solidification result, where the first solidification result is obtained by conducting a bucket test experiment on the test sub-experiment of the previous version to be released;

[0012] A test unit for using the first solidification result as a bucket in the test sub-experiment of the version to be released this time, and using this bucket as the initial value of the test sub-experiment of the version to be released this time to conduct a bucket test experiment on the test sub-experiment of the version to be released this time to obtain a second solidification result;

[0013] An update unit for updating the application according to the first solidification result and the second solidification result.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, where the executable instructions, when executed, cause the processor to execute any of the above methods.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including multiple application programs, the electronic device is caused to execute any of the above methods.

[0016] At least one of the above technical solutions adopted by the embodiments of the present application can achieve the following beneficial effects:

[0017] This application constructs a correlated parent-child test experiment. The correlated parent-child test experiment can include several test sub-experiments, each test sub-experiment corresponding to a version to be released. Each test sub-experiment contains multiple buckets, and each bucket corresponds to a sub-version of the version to be released. During testing, the solidified result obtained from the previous test sub-experiment is used as the initial value of the current test sub-experiment for bucket testing experiments, and the application is updated by combining the first solidified result of the previous test sub-experiment and the second solidified result of the current test sub-experiment. Without changing the overall architecture of the application to be released, this application enables the solidified content to still have the ability of gray-scale release by establishing correlated parent-child test experiments, and realizes version replacement without having to re-establish the overall test experiment. Moreover, since the solidified content occupies less storage resources and transmission resources, this application also greatly reduces the storage pressure and transmission pressure on the backend, saves a large amount of backend resources, and meets higher usage requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application, form a part of the present application, and the illustrative embodiments and descriptions thereof are used to explain the present application, and do not constitute an improper limitation to the present application. In the drawings:

[0019] Figure 1 FIG. shows a schematic flow chart of an application release method based on a bucket experiment according to an embodiment of the present application;

[0020] Figure 2-a FIG. shows a schematic structural diagram of a test experiment of an application according to an embodiment of the present application

[0021] Figure 2-b FIG. shows a schematic structural diagram of a test experiment of an application according to another embodiment of the present application;

[0022] Figure 3 FIG. shows a schematic diagram of assigning an initial value to a second test sub-experiment according to an embodiment of the present application;

[0023] Figure 4 FIG. shows a schematic structural diagram of a test experiment of an application according to still another embodiment of the present application;

[0024] Figure 5 FIG. shows a schematic flow chart of an application release method based on a bucket experiment according to another embodiment of the present application;

[0025] Figure 6 FIG. shows a schematic structural diagram of an application release device based on a bucket experiment according to an embodiment of the present application;

[0026] Figure 7Schematic diagram of the structure of an electronic device in an embodiment of the present application. Detailed implementation manners

[0027] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the scope of protection of the present application.

[0028] The following will describe in detail the technical solutions provided in each embodiment of the present application with reference to the drawings.

[0029] The concept of the present application is to construct nested and associated parent-child test experiments, and use the content that has been solidified through bucket test experiments in the application as the initial value of the new experiment to continue the experiment, so that the requirements for which the experiment has been completed can continuously conduct the next experiment without relying on the release project. When making changes to the application, there is no need to re-release. Directly through the parent-child experiment method, the ability of cyclic gray-scale change is given to change the experiment requirements.

[0030] Figure 1 The flowchart showing the application release method based on bucket experiments according to an embodiment of the present application is as follows. Figure 1 It can be seen that the present application at least includes steps S110 to S140:

[0031] Step S110: Construct a test sub-experiment for the version to be released this time in the test parent experiment of the application. Among them, the test sub-experiment includes multiple buckets, and each bucket corresponds to a sub-version of the version to be released this time.

[0032] First, briefly introduce the meanings of the professional terms that appear in the present application in the art.

[0033] Bucket experiment, also known as AB experiment, is mainly an experiment conducted around users. From a statistical sense, observe the feedback of users on different product designs, interaction experiences, and processes, so as to guide the improvement direction of the product. Bucket experiments are widely used for iterative optimization of Internet products.

[0034] Change: Change the original user logic or behavior through certain technical means.

[0035] Gray scale: Gradually let users obtain new changes, such as 1% -> 2% -> 5% -> 10%, etc., that is, control the influence range of the online change.

[0036] Parent-child experiment: Realize more functions through the way of experiment nesting.

[0037] In the prior art, when a release of an application is carried out in the manner of a bucket experiment, a validity period is usually set for the bucket experiment, such as 6 months. After the expiration of the validity period, the obtained bucket experiment results are determined as fixed values in the long term. This process is called solidification, and the solidified content does not have the ability of gray-scale change subsequently. That is, when new requirements appear and it is desired to modify the solidified content, the application can only be re-released, that is, only full-scale change can be made, and it does not have the gray-scale release function.

[0038] Therefore, the present application proposes an application release method based on a bucket experiment, which enables the solidified content to be gray-scale released again through the nested manner of a parent-child experiment.

[0039] First, a test experiment of the application to be released is constructed. When there are new changes in the application to be released, the new function changes can be realized by constructing a parent-child experiment. Specifically, when the application is first released, a test parent experiment of the application to be released and at least one test sub-experiment need to be constructed, where the test sub-experiment is nested in the test parent experiment; when the application is not first released, the test sub-experiment of the current version to be released can be directly constructed in the test parent experiment of the application. In the above test sub-experiment, there are multiple buckets, and each bucket corresponds to a sub-version of the version to be released. The sub-version can be understood as different versions designed in parallel for the version to be released.

[0040] Figure 2-a shows a schematic structural diagram of a test experiment of an application according to an embodiment of the present application. From Figure 2-a it can be seen that the test experiment of the application includes a test parent experiment 100 of the application. In the test parent experiment 100, there is a test sub-experiment, which is denoted as the first test sub-experiment 110. The first test sub-experiment 110 corresponds to the first version of the application. Among them, in the first test sub-experiment 110, there are two buckets, which are denoted as bucket 111 and bucket 112 respectively. Among them, bucket 111 corresponds to the first sub-version of the first version, and bucket 112 corresponds to the second sub-version of the first version. For example, if there is a button on a web page in the application, bucket 111 corresponds to the button being designed as red, and bucket 112 corresponds to the button being designed as green. When the application is first released, a test experiment as shown in but not limited to Figure 2-a can be constructed.

[0041] Figure 2-b shows a schematic structural diagram of a test experiment of an application according to another embodiment of the present application. From Figure 2-bIt can be seen that the test experiment of the application includes the test master experiment 100 of the application. In the test master experiment 100, there are two test sub-experiments, namely the first test sub-experiment 110 and the second test sub-experiment 120. The first test sub-experiment 110 corresponds to the first version of the application, and the second test sub-experiment 120 corresponds to the second version of the application. Each test sub-experiment contains two buckets. The two buckets of the first test sub-experiment 110 are denoted as bucket 111 and bucket 112, and the two buckets of the second test sub-experiment 120 are denoted as bucket 121 and bucket 122. Among them, the first version of the application can be understood as an existing version that has been released, and the second version of the application can be understood as a to-be-released version to be released this time.

[0042] It should be noted that usually, when releasing an application, if it is the first release, a test master experiment and at least one test sub-experiment need to be established. When updating an already released application later, that is, releasing a new version, when constructing the test experiment, only one test sub-experiment needs to be constructed in the test master experiment of the application. Figure 2-a and Figure 2-b Taking the test experiment of the application shown as an example, when releasing the application for the first time, that is, releasing the first version of the application, it is necessary to construct the test master experiment 100 and the first test sub-experiment 110 ( Figure 2-a ), and when releasing the application for the second time, that is, releasing the second version of the application, only the second test sub-experiment 120 ( Figure 2-b ) needs to be constructed. At this time, the overall architecture of the test master experiment 100 and the first test sub-experiment 110 already exist and do not need to be constructed again.

[0043] It should be noted that in some embodiments of the present application, if the second version is not substantially different from the first version, the second test sub-experiment 120 can have the same architecture as the first test sub-experiment 110. When constructing the second test sub-experiment 120, there is no need to rewrite the code at this time. The original architecture can be kept unchanged, and only the specific changed content needs to be simply replaced. For the content that needs to be replaced, it can be identified by, but not limited to, regular expressions and then replaced correspondingly. This method can quickly realize the construction of the test sub-experiment and greatly save the labor and time costs of constructing the test experiment.

[0044] It should be noted that the application described in the present application is generalized and can be a software, an application program (APP), or a combination of one or more pages of the front-end interface, etc.

[0045] In this embodiment, the following takes the to-be-released version as the second version as an example for description. That is, during the construction process, in the prediction master experiment 100 of the application, only the prediction sub-experiment 120 needs to be constructed.

[0046] Step S120: Obtain the first curing result, where the first curing result is obtained by performing a bucketing test experiment on the test sub-experiment of the last version to be released.

[0047] When releasing the second version, obtain the curing result obtained by performing a bucketing test experiment on the test sub-experiment of the last version to be released, that is, the bucketing test result of the first test sub-experiment 110 corresponding to the first version, and record this result as the first curing result.

[0048] The process of performing a bucketing test on the first test sub-experiment 110 can be briefly described as follows: After the first version of the application goes live, a bucketing test experiment can be performed on the first test sub-experiment 110 corresponding to the first version. For example, the access traffic can be guided to different buckets in the first version experiment according to a predetermined ratio, dyed, and then determine which bucket has a better result. For the specific traffic allocation method, that is, which access hits which bucket, it can refer to the prior art, or the following push-pull combination method can be adopted. Each time an application, such as an app, starts, the client will request bucketing information from the server ("pull" process, for the server); each time the gray scale ratio changes, the server will send a notification to notify the client to pull the response interface to obtain the latest bucketing data ("push" process, for the server); the server will calculate the bucket hit by the current client according to the gray scale ratio, so as to achieve traffic allocation.

[0049] After determining the first curing result, other buckets are eliminated. When there is more access traffic arriving at the application, all traffic will be guided to the cured bucket.

[0050] It should be noted here that performing a bucketing test experiment on the first test sub-experiment 110 is a step executed when the previous round changes, that is, the first curing result has been formed when the previous round changes. In the current round of changes, only the curing result of the previous round needs to be obtained. Here, it is recorded as the first curing result. Specifically, when obtaining the first curing result of the first test sub-experiment in the previous round, what is obtained is which bucket is cured. Here, it is assumed that the first curing result is bucket 112.

[0051] Step S130: Use the first curing result as a bucket in the test sub-experiment of the current version to be released, and use this bucket as the initial value of the test sub-experiment of the current version to be released to perform a bucketing test experiment on the test sub-experiment of the current version to be released, and obtain the second curing result.

[0052] The second test sub-experiment 120 can be understood as the change to occur. In order to make the already solidified content gray-scaled again, the obtained first solidification result is used as a bucket of the second test sub-experiment 120, and this bucket is used as the initial value of this test, that is, the initial value of the second test sub-experiment 120, and a bucket test experiment is carried out on the second test sub-experiment 120.

[0053] Specifically, after obtaining the first solidification result, one of the buckets in the second test sub-experiment 120 can be set to be the same as the content of the solidified bucket in the first solidification result, that is, either bucket 121 or bucket 122 is set to be the same as bucket 112. Please refer to Figure 3 , Figure 3 shows a schematic diagram of assigning an initial value to the second test sub-experiment according to an embodiment of the present application. From Figure 3 it can be seen that before and after assigning the initial value to the second test sub-experiment 120, bucket 121 in the second test sub-experiment 120 is transformed into bucket 112, the solidification result bucket of the previous round.

[0054] It should be noted that the assignment of the initial value to the second test sub-experiment mentioned here means that the designs of the functions, components, modules, etc. corresponding to bucket 121 are set to be the same as those of the functions, components, modules, etc. corresponding to bucket 112.

[0055] After assigning the initial value to the second test sub-experiment 120, a bucket test experiment can be carried out. At this time, all traffic is diverted to bucket 112 (which can also be denoted as bucket 121) of the second test sub-experiment. The traffic can be gradually diverted to other buckets according to a preset ratio for coloring, and then the second solidification result is determined. Among them, the second solidification result can be bucket 112 or bucket 122.

[0056] It should be noted that bucket 122 is the change to occur relative to bucket 112, but it does not exclude that the effect of bucket 122 is not as good as that of bucket 112. Therefore, the second solidification result may still be bucket 112.

[0057] Step S140: Update the application to be released according to the first solidification result and the second solidification result.

[0058] Finally, according to the first solidification result and the second solidification result, the relevant functions, components, interfaces, etc. of the application to be released are updated.

[0059] Assume that the first solidification result is bucket 112 and the second solidification result is also bucket 112. Then, there is no need to update the application to be released at this time. The application to be released is solidified into bucket 112, and the traffic is still diverted to bucket 112 after solidification.

[0060] Assume that the first curing result is bucket 112 and the second curing result is also bucket 122. At this time, update the application to be released. Specifically, solidify the application to be released into bucket 122, and redirect the traffic to bucket 122 after solidification.

[0061] When there are new changes again, repeat steps S110 to S114 to achieve gray-scale change again.

[0062] From Figure 1 the method shown, it can be seen that in this application, an associated parent-child test experiment is constructed. The associated parent-child test experiment may include several test sub-experiments. Each test sub-experiment corresponds to a version to be released. Each test sub-experiment includes multiple buckets, and each bucket corresponds to a sub-version of the version to be released. During the test, use the curing result obtained in the previous test sub-experiment as the initial value of the current test sub-experiment to conduct a bucket test experiment, and update the application by combining the first curing result of the previous test sub-experiment and the second curing result of the current test sub-experiment. Without changing the overall architecture of the application to be released, this application enables the solidified content to still have the gray-scale release ability by establishing an associated parent-child test experiment, and realizes the version replacement without having to re-establish the overall test experiment. Moreover, since the solidified content occupies less storage resources and transmission resources, this application also greatly reduces the storage pressure and transmission pressure on the backend, saves a large amount of backend resources, and meets higher usage requirements.

[0063] In some embodiments of this application, in the above method, when the application is first released, the method further includes: constructing a test mother experiment and at least one test sub-experiment for the application, where the test sub-experiment is nested in the test mother experiment, and the test sub-experiment includes multiple buckets; conducting a bucket test experiment on the test sub-experiment, and using the obtained curing result as the release of the application.

[0064] Please refer to Figure 2-a again. When the application is released for the second time, it is necessary to construct a test mother experiment. In the test mother experiment, only the first test sub-experiment 110 can be constructed, that is, only one test sub-experiment is constructed, and the first test sub-experiment 110 is nested in the test mother experiment 100. When the application is released, the process first reaches the test mother experiment 100, and then flows to the first test sub-experiment 110, and the first test sub-experiment 110 is executed.

[0065] In the first test sub-experiment 110, two or more buckets can be set. In this embodiment, two buckets are taken as an example, namely bucket 111 and bucket 112. The methods for conducting the bucket test experiment on the first test sub-experiment 110 include but are not limited to: guiding all the access traffic to one bucket of the first test sub-experiment 110 respectively; gradually splitting the access traffic into other buckets of the first test sub-experiment 110 according to a preset ratio; determining the solidified bucket of the first test sub-experiment 110 according to the coloring results of each bucket of the first test sub-experiment 110; and taking the obtained solidified bucket as the first solidification result.

[0066] Please refer to FIG. 2. In the prior art, when conducting a bucket experiment, the traffic is usually directly guided to multiple buckets respectively. Taking the first test sub-experiment 110 as an example, in the prior art, the traffic is usually directly guided to each bucket according to a certain ratio, such as 50% and 50%. This method has the disadvantages of unreasonable traffic splitting and insufficient coloring.

[0067] When conducting the bucket test experiment on the first test sub-experiment 110, first, 100% of the access traffic can be guided to bucket 111 of the first test sub-experiment 110. At this time, the traffic in bucket 111 is 100%, and the traffic in bucket 112 is 0%; then gray-scaling is performed, that is, the access traffic is gradually distributed between bucket 111 and bucket 112 according to a preset ratio, so that the traffic in bucket 111 gradually changes from 100% to 50%, and the traffic in bucket 112 gradually changes from 0% to 50%. Then coloring is performed. The process of traffic splitting is usually carried out step by step. For example, in the first period of time, the traffic in bucket 111 is 100%, and the traffic in bucket 112 is 0%. Then the traffic in bucket 111 is made 90%, and the traffic in bucket 112 is made 10%. Then the traffic in bucket 111 is made 80%, and the traffic in bucket 112 is made 20%, and so on, until the traffic in both bucket 111 and bucket 112 reaches 50%. Then the effects are compared. By adopting the method of gradual gray-scaling, the traffic splitting method is more reasonable, the coloring is sufficient, and the obtained indicators are more accurate.

[0068] There are data points set on the server side, and the client will report the experimental data of the bucket to the data points on the server side. This process is called coloring. According to the coloring results, the impact on the evaluation indicators can be seen, so as to determine which bucket has a better effect. For example, if the evaluation indicator is the conversion rate, and the conversion rate of bucket 111 is higher than that of bucket 112, it means that the effect of bucket 111 is better. Among them, bucket 111 is used as the first solidification result.

[0069] In some other embodiments of the present application, the number of the test sub-experiments is multiple, and each test sub-experiment corresponds to a version to be released; the method further includes: in response to a directed instruction for the version to be released, performing the test sub-experiment of the version to be released corresponding to the directed instruction.

[0070] In some scenarios, when building a test experiment, multiple test sub-experiments can be directly built in the test mother experiment as needed, and each test sub-experiment corresponds to a version to be released. When releasing an application, the corresponding test sub-experiment can be performed according to the directed instruction for the version to be released.

[0071] Taking Figure 2-b the described test experiment as an example, assume that the first test sub-experiment 110 and the second test sub-experiment 120 are both built when the application is first released. The first test sub-experiment 110 corresponds to the first version, and the second test sub-experiment 120 corresponds to the second version, and the first version and the second version are parallel. When releasing the application, the version to be released can be specified. If you want to release the first version, you can directly perform a directed designation on the first test sub-experiment 110, and the process directly flows to the first test sub-experiment 110. After the first test sub-experiment 110 is executed, it is fully solidified into a bucket of the first test sub-experiment, and the first test sub-experiment 110 is recorded as the first solidification result. If after the first version is released and you want to release the second version, you can perform a directed designation on the second test sub-experiment 120. When the second test sub-experiment 120 is executed, specifically obtain the first solidification result, use the first solidification result as a bucket in the test sub-experiment of the current version to be released, use this bucket as the initial value of the test sub-experiment of the current version to be released, and perform a bucket test experiment on the test sub-experiment of the current version to be released to obtain the second solidification result; update the application according to the first solidification result and the second solidification result. Thus, the already solidified content again has the ability of gray scale.

[0072] Similarly, if when the application is first released, if you want to directly release the second version, you can directly perform a directed designation on the second test sub-experiment 120, and the process directly flows to the second test sub-experiment 120. If after the second version is released and you want to release the first version, you can perform a directed designation on the first test sub-experiment 110 again.

[0073] In an actual application scenario, usually a version to be released application includes multiple functions. Therefore, in some embodiments of the present application, the application to be released can be divided into multiple components or modules, and multiple component test experiments can be respectively set in the first test sub-experiment 110 and the second test sub-experiment 120, and these components or modules are respectively tested based on each component test experiment.

[0074] In some embodiments of the present application, constructing a test sub-experiment for the version to be released in the test mother experiment of the application includes: dividing the application into multiple components; constructing component test experiments for each component of the version to be released, each of the component test experiments including multiple buckets with the same number, and there is a one-to-one correspondence between the buckets of different component test experiments, and multiple buckets located in different component test experiments with a corresponding relationship form a group of buckets.

[0075] In some embodiments of the present application, in an actual application scenario, an application is composed of multiple components or modules. Therefore, during experiments, an application is usually divided into multiple components, such as Figure 4 shown Figure 4 shows a schematic structural diagram of a test experiment for an application to be released according to another embodiment of the present application. From Figure 4 it can be seen that the application is divided into three components, denoted as Component 1, Component 2, and Component 3 respectively. For each component, a component test experiment is constructed. That is, three component test experiments can be set in the first test sub-experiment 110. The first test sub-experiment 110 includes a first component test experiment A, a second component test experiment B, and a third component test experiment C. Among them, each component test experiment corresponds to a component in the application. The first component test experiment A corresponds to Component 1, the second component test experiment B corresponds to Component 2, and the third component test experiment C corresponds to Component 3. Multiple buckets can be set in each component test experiment, and each bucket corresponds to a different sub-version of the component. Different sub-versions can be understood as different designs of a component. Taking the first component test experiment A as an example, bucket A1 and bucket A2 correspond to two different sub-versions of Component 1.

[0076] It should be noted that the splitting method of components between different versions is the same. That is, if in the first version, the application is split into the form of the combination of Component 1, Component 2, and Component 3, then in the second version, the application is still split into these 3 components. Just a new version appears. In the second version, it can still be understood that the application is in the form of the combination of Component 1, Component 2, and Component 3. It can be understood that different versions of an application include the same number of components in one-to-one correspondence. Therefore, the test sub-experiments of different versions have the same number of component test experiments, and there is a one-to-one correspondence between the component test experiments corresponding to the same component in the test sub-experiments of different versions.

[0077] Therefore, based on the above-mentioned segmentation and construction in the first test sub-experiment 110, three component test experiments can also be set up in the second test sub-experiment 120, namely the fourth component test experiment D, the fifth component test experiment E, and the sixth component test experiment F. That is, the first component test experiment A has a corresponding relationship with the fourth component test experiment D, and both correspond to component 1; the second component experiment B and the fifth component test experiment E have a corresponding relationship, and both correspond to component 2; the third component test experiment C and the sixth component test experiment F have a corresponding relationship, and both correspond to component 3. The fourth component test experiment D can be understood as a change to the first component test experiment A, the fifth component test experiment E can be understood as a change to the second component test experiment B, and the sixth component test experiment F can be understood as a change to the third component test experiment C.

[0078] And in one test sub-experiment, each component test sub-experiment has the same number of buckets, and there is a one-to-one correspondence between the buckets of different component test experiments. Multiple buckets located in different component test experiments with a corresponding relationship form a group of buckets. Please refer to Figure 4 again. Taking the first test sub-experiment 110 as an example, it includes three component test experiments. Each component test experiment contains two buckets. The buckets of the first component test experiment A are respectively denoted as bucket A1 and bucket A2; the buckets of the second component test experiment B are respectively denoted as bucket B1 and bucket B2; the buckets of the third component test experiment C are respectively denoted as bucket C1 and bucket C2. Among them, bucket A1, bucket B1, and bucket C1 form a group of buckets; bucket A2, bucket B2, and bucket C2 form a group of buckets. Taking the second test sub-experiment 120 as an example, it also includes three component test experiments. Each component test experiment contains three buckets. The buckets of the fourth component test experiment D are respectively denoted as bucket D1, bucket D2, and bucket D3; the buckets of the fifth component test experiment E are respectively denoted as bucket E1, bucket E2, and bucket E3; the buckets of the sixth component test experiment F are respectively denoted as bucket F1, bucket F2, and bucket F3. Among them, bucket D1, bucket E1, and bucket F1 form a group of buckets; bucket D2, bucket E2, and bucket F2 form a group of buckets; bucket D3, bucket E3, and bucket F3 form a group of buckets. When making changes, it is only necessary to synchronously conduct bucket experiment tests on the component test experiments of each component.

[0079] In some embodiments of the present application, in the above method, the first solidified result is used as a bucket in the test sub-experiment of the version to be released, and the bucket is used as the initial value of the test sub-experiment of the version to be released. The test sub-experiment of the version to be released is subjected to a bucket test experiment to obtain a second solidified result, including: setting any component bucket in the different component test experiments in the test sub-experiment of the version to be released to be consistent with the first solidified result; directing all access traffic to a component bucket consistent with the first solidified result; gradually diverting the access traffic to other component buckets of the test sub-experiment of the version to be released according to a preset ratio; and determining the second solidified result based on the coloring results of each component bucket.

[0080] Please refer to Figure 4 , assuming that the version to be released this time is the second version, and then the first test sub-experiment 110 is tested by bucketing, and the solidified result is a bucket group consisting of bucket A1, bucket B1 and bucket C1. When the second test sub-experiment 120 is tested by bucketing, the initial value of the second test sub-experiment 120 is first assigned. Specifically, any group of buckets in the second test sub-experiment 120 is set to be consistent with the first solidified result, from Figure 4 It can be seen that the second test sub-experiment 120 is set with 3 sub-groups, and any one of the groups can be set to be consistent with the first solidification result, that is, consistent with the bucket group composed of bucket A1, bucket B1 and bucket C1. Assume that the bucket group composed of bucket D1, bucket E1 and bucket F1 is set to be consistent with the bucket group composed of bucket A1, bucket B1 and bucket C1, and then the bucket test experiment can be carried out.

[0081] When conducting a bucket test experiment, firstly, 100% of the access traffic is directed to the bucket group consisting of bucket D1, bucket E1 and bucket F1, which is consistent with the bucket group consisting of bucket A1, bucket B1 and bucket C1. That is to say, the second test sub-experiment 120 is conducted based on the first solidification result, so that the solidified content has the ability to change grayscale again. The subsequent process is the same as the bucket experiment process of the first test sub-experiment 110, that is, gradually making the traffic in the three bucket groups reach a preset proportion, such as one-third each, and then compare the effects. Here, it is assumed that after the bucket experiment, the bucket group consisting of bucket D2, bucket E2 and bucket F2 has the best effect, then the second version experiment is solidified to the bucket group consisting of bucket D2, bucket E2 and bucket F2 as the solidification result of the second test sub-experiment 120. It should be noted that the bucket test experiments for multiple component test experiments are synchronized, but the effect can be for the same bucket group or across bucket groups. For example, in this embodiment, the second solidification result can be a combination of bucket D1, bucket E2 and bucket F3.

[0082] In some embodiments of the present application, updating the application according to the first curing result and the second curing result includes: comparing the consistency of the curing buckets in the first curing result and the curing buckets in the second curing result; if they are consistent, updating the application according to the first curing result or the second curing result; if they are inconsistent, updating the application according to the second curing result.

[0083] In the step of updating the application, it can be carried out according to the consistency between the first curing result and the second curing result. Specifically, compare whether the curing buckets in the first curing result are consistent with the curing buckets in the second curing result. If the two are consistent, it means that the effect of the change in the version to be released this time is not as good as the existing version. At this time, there is no need to update the application and it can remain unchanged. If the curing buckets in the first curing result are inconsistent with the curing buckets in the second curing result, it means that the effect of the change in the version to be released this time is better than the existing technology. Then, update the existing version of the application according to the current curing result, that is, the second curing result, to achieve the replacement of the version.

[0084] In some embodiments of the present application, in the above method, updating the application according to the first curing result and the second curing result includes: comparing the consistency of each curing bucket in the first curing result and each curing bucket in the second curing result; if a curing bucket in the first curing result is consistent with the corresponding curing bucket in the second curing result, keep the component corresponding to this curing bucket in the application unchanged; if a curing bucket in the first curing result is inconsistent with the corresponding curing bucket in the second curing result, update the component corresponding to this curing bucket according to the second curing result.

[0085] In the case where the application is divided into multiple components, during the comparison of the first curing result and the second curing result, it is necessary to compare them one by one according to the corresponding component test experiments in the test sub-experiments of the two versions. Still taking Figure 4Taking the illustrated embodiment as an example, when comparing the first curing result with the second curing result, the curing bins obtained in the first component test experiment A are compared with the curing bins obtained in the fourth component test experiment D. Suppose the curing bin obtained in the first component test experiment A is bin A1, and the curing bin obtained in the fourth component test experiment D is bin D1 which is set to be the same as bin A1. Then it is confirmed that for component 1, the first curing result is consistent with the second curing result and no change is required. Suppose the curing bin obtained in the first component test experiment A is bin A1, and the curing bin obtained in the fourth component test experiment D is bin D2. Since bin D2 is not set the same as bin A1, it is confirmed that for component 1, the first curing result is inconsistent with the second curing result, and component 1 needs to be changed. Specifically, the existing version of component 1 is updated to the second version of component 1.

[0086] And so on, the same method applies to component 2 and component 3. It should be noted that the comparison of the curing results of each component and the update process of the components can be carried out synchronously, such as multi-threaded parallel, or asynchronously, such as single-threaded serial. This application does not make a limitation in this regard, but for the sake of improving efficiency, it is recommended to adopt the synchronous method.

[0087] Table 1 shows the results of application release according to an embodiment of the present application. It can be seen from Table 1 that the application contains 3 components. For component 1, the first component test experiment A and the fourth component test experiment D correspond to component 1. The first curing result of the first component test experiment A is bin A1. After setting bin D1 to be the same as bin A1 (bin D1 = bin A1) and performing the bin test experiment, the obtained second curing result is bin D2. Then for component 1, its second curing result is inconsistent with the first curing result and needs to be updated. After the update, component 1 is replaced with the change corresponding to bin D2. The situation of component 3 is the same as that of component 1 and will not be elaborated in detail.

[0088] For component 2, the second component experiment B and the fifth component test experiment E correspond to component 2. The first curing result of the second component test experiment B is bin B2. After setting bin E1 to be the same as bin B2 (bin B2 = bin E1) and performing the bin test experiment, the obtained second curing result is bin E1. Then for component 2, its second curing result is consistent with the first curing result. Therefore, no update is required.

[0089] Table 1

[0090]

[0091] Figure 5 shows a schematic flow diagram of an application release method based on bin experiments according to another embodiment of the present application. From Figure 5It can be seen that in this embodiment, in the test mother experiment, it involves the process of an application release version 1.0, release version 2.0, and release version 3.0. From Figure 5 It can be seen that for the processes of release version 1.0, release version 2.0, and release version 3.0, test sub-experiments A, D, and G are respectively constructed. Two buckets are set in test sub-experiment A, test sub-experiment D, and test sub-experiment G, and each bucket corresponds to a different sub-version of its corresponding version.

[0092] In release version 1.0, first, 100% of the access traffic is guided to bucket A1 of experiment A, and then gradually a part is diverted to bucket A2 of experiment A. After dyeing, it is assumed to be solidified in bucket A2.

[0093] In release version 2.0, test sub-experiment D is constructed. First, one bucket in test sub-experiment D is set to be the same as bucket A2 of experiment A, and 100% of the access traffic is guided to this bucket (this bucket can be denoted as bucket A2 or also called bucket D1), and then gradually a part is diverted to bucket D2 of experiment D. After dyeing, it is assumed to be solidified in bucket D2.

[0094] Similar to release version 2.0, in release version 3.0, test sub-experiment G is constructed. First, one bucket in test sub-experiment G is set to be the same as bucket D2 of experiment D, and 100% of the access traffic is guided to this bucket (this bucket can be denoted as bucket D2 or also called bucket G1), and then gradually a part is diverted to bucket G2 of experiment G. After dyeing, it is solidified in bucket G2. After solidification, 100% of the access traffic can be guided to bucket G2.

[0095] Figure 6 The structure diagram of an application release device based on a bucket experiment according to an embodiment of the present application is shown. From Figure 6 It can be seen that the device 600 includes:

[0096] A construction unit 610, configured to construct a test sub-experiment of the current version to be released in the test mother experiment of the application, where the test experiment sub includes multiple buckets, and each bucket corresponds to a sub-version of the current version to be released;

[0097] An acquisition unit 620, configured to acquire a first solidification result, where the first solidification result is obtained by performing a bucket test experiment on the test sub-experiment of the previous version to be released;

[0098] A test unit 630 is configured to use the first curing result as a bucket in a sub-experiment of the test for the version to be released this time, and use this bucket as the initial value of the sub-experiment of the test for the version to be released this time to perform a bucket test experiment on the sub-experiment of the test for the version to be released this time, so as to obtain a second curing result;

[0099] An update unit 640 is configured to update the application according to the first curing result and the second curing result.

[0100] In some embodiments of the present application, in the above device, a construction unit 610 is further configured to construct a test mother experiment and at least one test sub-experiment of the application when the application is released for the first time, wherein the test sub-experiment is nested in the test mother experiment, and the test sub-experiment includes multiple buckets; the test unit 630 is further configured to perform a bucket test experiment on the test sub-experiment, and use the obtained curing result as the release of the application.

[0101] In some embodiments of the present application, in the above device, the number of the test sub-experiments is multiple, and each test sub-experiment corresponds to a version to be released; the test unit 630 is further configured to execute the test sub-experiment of the version to be released corresponding to the orientation instruction in response to the orientation instruction for the version to be released.

[0102] In some embodiments of the present application, in the above device, the construction unit 610 is configured to divide the application into multiple components; construct component test experiments for each component of the version to be released, each component test experiment includes the same number of multiple buckets, and there is a one-to-one correspondence between the buckets of different component test experiments, and multiple buckets located in different component test experiments with a corresponding relationship form a group of buckets.

[0103] In some embodiments of the present application, in the above device, the test unit 630 is configured to set any group of buckets in different component test experiments in the test sub-experiment of the version to be released this time to be consistent with the first curing result; direct all access traffic to a group of buckets consistent with the first curing result; gradually split the access traffic into other component buckets of the test sub-experiment of the version to be released this time according to a preset ratio; determine the second curing result according to the staining results of each component bucket.

[0104] In some embodiments of the present application, in the above device, the update unit 640 is configured to compare the consistency of the cured buckets in the first curing result with the cured buckets in the second curing result; if they are consistent, update the application according to the first curing result or the second curing result; if they are inconsistent, update the application according to the second curing result.

[0105] In some embodiments of the present application, in the above-mentioned apparatus, the updating unit 640 is configured to compare the consistency of each curing bin in the first curing result with each curing bin in the second curing result; if a curing bin in the first curing result is consistent with the corresponding curing bin in the second curing result, the component corresponding to this curing bin in the application is kept unchanged; if a curing bin in the first curing result is inconsistent with the corresponding curing bin in the second curing result, the component corresponding to this curing bin is updated according to the second curing result.

[0106] It should be noted that the above-mentioned application release device based on the bucket experiment can implement the corresponding application release method based on the bucket experiment one by one, and will not be elaborated here one by one.

[0107] Figure 7 It is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 7 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include internal memory, such as high-speed random access memory (Random-Access Memory, RAM), and may also include non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required by other applications.

[0108] The processor, network interface, and memory can be interconnected through an internal bus, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 7 only a bidirectional arrow is used in

[0109] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include internal memory and non-volatile memory, and provide instructions and data to the processor.

[0110] The processor obtains the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an application publishing device based on the bucketing experiment at the logical level. The processor executes the program stored in the memory and is specifically used to execute any of the foregoing methods.

[0111] As described in this application Figure 6 The method executed by the application publishing device based on the bucketing experiment disclosed in the embodiments shown above can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor or by instructions in software form. The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute each method, step, and logic block diagram disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of this application can be directly embodied as being executed and completed by the hardware decoding processor, or by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, etc. This storage medium is located in the memory, and the processor obtains the information in the memory and combines its hardware to complete the steps of the above method.

[0112] The electronic device can also execute Figure 6 the method executed by the application publishing device based on the bucketing experiment and implement the functions of the application publishing device based on the bucketing experiment in Figure 6 the embodiments shown. The embodiments of this application will not be elaborated here.

[0113] The embodiments of this application also propose a computer-readable storage medium. The computer-readable storage medium stores one or more programs. The one or more programs include instructions that, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 6 the method executed by the application publishing device based on the bucketing experiment in the embodiments shown and is specifically used to execute any of the foregoing methods.

[0114] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0115] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0116] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are performed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0118] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0119] The memory may include non-permanent memory in the form of computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory is an example of computer-readable media.

[0120] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0121] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0122] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, system or computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] The above description is only for the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An application release method based on a bucketing experiment, characterized in that, it includes: Construct a test sub-experiment for the version to be released this time in the test mother experiment of the application, where the test sub-experiment includes multiple buckets, and each bucket corresponds to a sub-version of the version to be released this time; Obtain the first solidification result, where the first solidification result is obtained through a bucketing test experiment on the test sub-experiment of the previous version to be released; Use the first solidification result as a bucket in the test sub-experiment of the version to be released this time, and use this bucket as the initial value of the test sub-experiment of the version to be released this time to conduct a bucketing test experiment on the test sub-experiment of the version to be released this time, and obtain the second solidification result; Update the application according to the first solidification result and the second solidification result, specifically including: comparing the consistency of the solidified buckets in the first solidification result with the solidified buckets in the second solidification result; If they are consistent, update the application according to the first solidification result or the second solidification result; if they are inconsistent, update the application according to the second solidification result.

2. The method according to claim 1, characterized in that, When first releasing the application, the method further includes: Construct a test mother experiment and at least one test sub-experiment of the application, where the test sub-experiment is nested in the test mother experiment, and the test sub-experiment contains multiple buckets; Conduct a bucketing test experiment on the test sub-experiment, and use the obtained solidification result as the release of the application.

3. The method according to claim 2, characterized in that, The number of the test sub-experiments is multiple, and each test sub-experiment corresponds to a version to be released; The method further includes: In response to a directional instruction for the version to be released, execute the test sub-experiment of the version to be released corresponding to the directional instruction.

4. The method according to claim 1, characterized in that, The constructing a test sub-experiment for the version to be released this time in the test mother experiment of the application includes: Divide the application into multiple components; Construct component test experiments for each component of the version to be released. Each component test experiment includes the same number of multiple buckets, and there is a one-to-one correspondence between the buckets of different component test experiments. Multiple buckets located in different component test experiments with a corresponding relationship form a group of buckets.

5. The method according to claim 4, characterized in that, The using the first solidification result as a bucket in the test sub-experiment of the version to be released this time, and using this bucket as the initial value of the test sub-experiment of the version to be released this time to conduct a bucketing test experiment on the test sub-experiment of the version to be released this time, and obtaining the second solidification result includes: Set any group of buckets in different component test experiments in the test sub-experiment of the version to be released this time to be consistent with the first solidification result; Direct all access traffic to a group of buckets that is consistent with the first solidification result; Gradually split the access traffic into other component buckets of the test sub-experiment of the version to be released this time according to a preset ratio; Determine the second curing result according to the dyeing results of each component bucket.

6. The method according to claim 1, wherein, updating the application according to the first curing result and the second curing result more specifically includes: comparing the consistency of each curing bucket in the first curing result with each curing bucket in the second curing result; if a curing bucket in the first curing result is consistent with the corresponding curing bucket in the second curing result, keep the component corresponding to this curing bucket in the application unchanged; if a curing bucket in the first curing result is inconsistent with the corresponding curing bucket in the second curing result, update the component corresponding to this curing bucket according to the second curing result.

7. An application release device based on a bucket experiment, wherein, the device includes: a construction unit, configured to construct a test sub-experiment of the version to be released this time in the test mother experiment of the application, wherein the test experiment sub includes a plurality of buckets, and each bucket corresponds to a sub-version of the version to be released this time; an acquisition unit, configured to acquire a first curing result, wherein the first curing result is obtained by performing a bucket test experiment on the test sub-experiment of the previous version to be released; a test unit, configured to use the first curing result as a bucket in the test sub-experiment of the version to be released this time, and use this bucket as the initial value of the test sub-experiment of the version to be released this time to perform a bucket test experiment on the test sub-experiment of the version to be released this time, and obtain a second curing result; an update unit, configured to update the application according to the first curing result and the second curing result, specifically configured to compare the consistency of the curing buckets in the first curing result with the curing buckets in the second curing result; if they are consistent, update the application according to the first curing result or the second curing result; if they are inconsistent, update the application according to the second curing result.

8. An electronic device, including: a processor; and a memory arranged to store computer-executable instructions, and the executable instructions, when executed, cause the processor to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute the method according to any one of claims 1 to 6.

Images