Method and device for testing, processor and computing equipment

Abstract

The embodiment of the invention relates to a method and device for testing, a processor and computing equipment. The method comprises the following steps: acquiring a plurality of test traffics for a target hierarchy test from a plurality of initial test traffics, wherein the target hierarchy comprises a plurality of tests; the method further comprises the steps that the multiple test flows are distributed to the multiple tests on the basis of identification information of the multiple test flows, and the identification information is stored in the client side and used for representing whether the test flows participate in historical tests or not and the participated test versions. In addition, the method further comprises the step of determining a corresponding test result by determining the feedback of the target test flow for the target test version in the target test based on the distribution result.

Description

Technical Field

[0001] The present disclosure relates to the field of software testing, and more particularly to a method, apparatus, processor, and computing device for testing. Background Art

[0002] During the software development stage such as application development, multiple version schemes are usually designed. To determine the better scheme among multiple version schemes, each version scheme can be tested. Among them, A / B testing is to design two (A / B) or more (A / B / n) versions for a certain software. In the same time dimension, different visitor groups with the same or similar composition are randomly allowed to access these versions respectively. By collecting the user experience data of each group, the version that performs better and is more popular among users can be finally analyzed. During the A / B testing process, it is necessary to split the visitor groups so that users within the same group can randomly access the various test versions of the same software. Therefore, how to split users has become the key to improving the test efficiency and test accuracy. Summary of the Invention

[0003] Embodiments of this specification provide a method, apparatus, processor, and computing device for testing.

[0004] In a first aspect of this specification, a method for testing is provided. The method includes obtaining multiple test traffic for target-level testing from multiple initial test traffic, where the target level includes multiple trials. The method further includes allocating the multiple test traffic to multiple trials based on the identification information of the multiple test traffic, and the identification information is used to characterize whether the test traffic has participated in historical testing and the test versions participated in. In addition, the method further includes determining the corresponding test result based on the allocation result by determining the feedback of the target test traffic for the target test version in the target trial.

[0005] In a second aspect of this specification, a device for testing is provided. The device includes a traffic acquisition module configured to obtain multiple test traffic for target-level testing from multiple initial test traffic, where the target level includes multiple trials. The device further includes a traffic allocation module configured to allocate the multiple test traffic to multiple trials based on the identification information of the multiple test traffic, and the identification information is used to characterize whether the test traffic has participated in historical testing and the test versions participated in. In addition, the device further includes a test result determination unit configured to determine the corresponding test result based on the allocation result by determining the feedback of the target test traffic for the target test version in the target trial.

[0006] In a third aspect of this specification, a processor is provided. The processor is used to execute the method according to the first aspect of this specification.

[0007] In a fourth aspect of the present specification, a computing device is provided. The computing device includes a memory. The computing device further includes a processor according to the third aspect of the present specification.

[0008] It should be understood that the content described in the Summary of the Invention section is not intended to limit the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent. In the drawings, the same or similar reference numerals denote the same or similar elements, where:

[0010] Figure 1 A schematic diagram of an example environment in which some embodiments of the present specification can be implemented is shown;

[0011] Figure 2 A schematic diagram of the work flow of each execution entity during testing for some embodiments of the present specification is shown;

[0012] Figure 3 A schematic diagram of splitting test traffic for some embodiments of the present specification is shown;

[0013] Figure 4 A flowchart of a method 400 for testing for some embodiments of the present specification is shown;

[0014] Figure 5 A schematic diagram of distributing test traffic for some embodiments of the present specification is shown;

[0015] Figure 6 A schematic diagram of distributing test traffic for some other embodiments of the present specification is shown;

[0016] Figure 7 A schematic diagram of the process of test traffic distribution provided by some embodiments of the present specification is shown; and

[0017] Figure 8 A schematic block diagram of a computing device for some embodiments of the present specification is shown.

[0018] In all the drawings, the same or similar reference numerals represent the same or similar elements. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and its like should be understood as an open inclusion, that is, "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. There may also be other explicit and implicit definitions hereinafter.

[0021] As described above, after the server receives the access request from the user, it is crucial to correctly assign the user to a certain test version of the application. In the related art, different users can be assigned to different test versions according to the experiment configuration information pre-set by users such as operation personnel. However, this method requires the experiment configuration information to be stored in the server, which will cause additional consumption of server resources (such as CPU, memory) due to the additional query steps for the newly added server while increasing the burden on the server.

[0022] For this reason, the embodiments of this specification provide a testing method, which can select multiple test traffic for testing the target layer from multiple initial test traffic, and other traffic can be used to test other layers. Since a layer contains multiple experiments and an experiment contains multiple test versions, the test version to which the test traffic should be assigned in the current test can be determined according to the allocation result of the test traffic in the historical test. In this way, based on the allocation result, the feedback of the test traffic for the test version can be used to determine the final test result.

[0023] In this way, since the allocation result of the test traffic in the previous test is stored on the client, only the resources of the client are needed to determine the test versions corresponding to different test traffic, and the server only needs to send the corresponding test version or the page data corresponding to the test version to the client, thereby effectively reducing the data processing volume and data storage volume of the server and reducing the resource consumption of the server. At the same time, by adopting the method of hierarchical traffic splitting for the test experiment, it can be ensured that the experiments between different layers do not interfere with each other, and the experiments in the same layer share the test traffic, thereby improving the test efficiency and reducing the test cost.

[0024] Figure 1FIG. shows a schematic diagram of an example environment 100 in which multiple embodiments of the present disclosure can be implemented. As Figure 1 shown, in environment 100, it includes clients 102 (such as client 102-1, client 102-2, and client 102-3) and a server 106 connected to clients 102 via a network (such as a wireless communication link or an optical fiber cable, etc.). The server 106 can specifically be a single server, or can include several servers, or is a server cluster formed by several servers. It can be understood that the server 106 can be a server providing various services, for example, a backend server for testing application programs installed on the client.

[0025] In some embodiments, the client can be an electronic device installed with an application program, including but not limited to a mobile smart phone, a computer (including a laptop and a desktop computer), a tablet electronic device, a personal digital assistant, or a smart wearable device, etc. Among them, the application program can be a payment application, a social application, a game application, a short video application, a video application, a music application, a shopping application, etc. In other embodiments, the client 102 can also be software running on any of the above devices, such as an Alipay client, a Taobao client, etc. The form of the client here is not specifically limited. Similarly in the following without special statement.

[0026] It should be noted that the testing method provided by the embodiments of this specification is generally executed by the server 104. In environment 100, user A can access the application program A deployed on client 102-1 through the interaction operation provided by the client. The client 102-1 can generate a corresponding access request based on the user's interaction operation and send the access request to the server 104. It can be understood that the server 104 can use the access request as test traffic and determine the test version allocated to the test traffic according to the identification information carried by the test traffic. Among them, the identification information can be stored on the client 102-1. That is to say, the identification information carried by the test traffic is held by the client 102. The server only needs to correctly and reasonably split the test traffic according to the reference allocation result provided by the client 102.

[0027] Figure 2 FIG. shows a schematic diagram of the working processes of each execution entity during the testing of some embodiments of this specification. As Figure 2As shown, users such as operation staff can configure the test experiment of the application according to actual test requirements or operation requirements to generate configuration information. Among them, the configuration information can include the duration of the test experiment, the traffic splitting strategy of the test experiment, the information corresponding to multiple test versions (such as the number of test versions), etc. For example, the operation staff can input corresponding configuration parameters through the interaction interface of the client 2002, and the client 2002 can generate corresponding configuration information based on the user input. This configuration information can be sent by the client 2002 to the server 2004.

[0028] In block 202, the server 2004 can obtain this configuration information, for example, obtain this configuration information from the client 2002 or other configuration servers. In block 204, the client 2002 can receive the user's interaction operations (such as clicks, swipes, etc.) to generate an access request or an activation request for the application, and send this access request or activation request to the server 2004. For example, if the user wants to access a certain application of interest on the client 2002, the user can initiate a trigger operation for this application on the client 2002. For example, click on the icon corresponding to this application on the client 2002. The client 2002 will receive and respond to the user's trigger operation for the target page to generate a corresponding access request. It can be understood that an access request can be regarded as a test traffic. This test traffic can carry identification information, and this identification information can be used to indicate to which test version of which experiment the test traffic was assigned in the previous test.

[0029] As Figure 2 shown, in block 206, the server 2004 can complete the allocation of this test traffic according to the identification information, that is, the server 2004 presents the test version corresponding to the test traffic to the user to conduct a test on the user for this test version. For example, the server 2004 can send the page data corresponding to this test version to the client 2002, and the display screen of the client 2002 displays the page data. In this way, the user can perform corresponding operations (such as browsing, favoriting, switching, etc.) on this page on the client 2002, and the client 2002 can automatically collect the user behavior data of the user for the test version and send this user behavior data to the server 2004. After receiving the user behavior data, the server 2004 can analyze the user behavior data to determine the corresponding test result.

[0030] At block 208, the server 2004 may send the test result to the client 2002. Among them, the test result may be which test version is more popular among users and which test version provides the best user experience for users, that is, which test version is the final version of the application. At block 210, the client 2002 may display the test result. For example, the client 2002 may display the test result to the operation staff, and the operation staff may perform subsequent evaluation and processing based on the test result.

[0031] Figure 3 The figure shows a schematic diagram of shunting test traffic for some embodiments of this specification. As Figure 3 shown, the test may include multiple levels of tests, such as including level 1 and level 2. The tests between different levels do not affect each other, and the tests between multiple levels can be executed in parallel, and the test traffic for different levels of tests satisfies orthogonality. The test traffic for level 1 test is 100%, and the test traffic for level 2 test can also be 100%.

[0032] As Figure 3 shown, a level may contain multiple trials, and the test trials within the same level can shunt the test traffic according to the shunting strategy. For example, multiple trials within the same level can share 100% of the test traffic according to a preset ratio. Trial 1 of level 1 can shunt 10% of the test traffic, and trial 2 of level 1 can shunt 10% of the test traffic. This test method similar to pipeline shunting can not only avoid the mutual influence and interference between different trials, but also improve the test efficiency through parallel testing.

[0033] At the same time, a trial may include multiple test versions. For example, trial 1 may include four versions, such as the original version, test version A, test version B, and test version C. The purpose is to test which test version can bring a better user experience to the test users compared to the original version. Multiple versions within a trial can also share 10% of the test traffic according to the shunting strategy (preset shunting ratio). For example, the original version can shunt 2.5% of the test traffic, and similarly, test version A, test version B, and test version C can each shunt 2.5% of the test traffic.

[0034] In this way, during the process of shunting test traffic, different test traffic can be allocated to different "pipelines" (trials) or different test versions according to the shunting strategy. Thus, different identification information can be assigned to different test traffic based on the allocation results. Among them, the identification information can include a participation trial mark and a participation version mark. Among them, the participation trial mark can include whether the test traffic is allocated to the corresponding trial and the label of the allocated trial. The participation version mark can include whether the test traffic is allocated to the corresponding test version and the label of the allocated test version. It should be noted that the test traffic within one level shares a set of marking methods. For example, the labels of multiple trials within level 1 are the same set of labels, and the labels within one trial are the same set of labels. Level 1 cannot identify the identification information of the test traffic in level 2.

[0035] In this way, the server can assign different allocation labels to different test traffic and send the allocation labels to the client for storage by the client. On the one hand, this can reduce the storage pressure on the server. On the other hand, the server allocates the test traffic to the correct test version according to the allocation label carried by the test traffic, so as to ensure the effectiveness of the test throughout the test cycle.

[0036] Figure 4 The flowchart of method 400 for testing according to some embodiments of the present specification is shown. In some embodiments, in the exemplary environment 100 shown in Figure 1 method 400 may be executed by server 104. It should be understood that although the following content is described with server 104 as the execution subject, method 400 may also be executed by other devices. Method 400 may further include additional actions not shown and / or may omit the actions shown, and the scope of this specification is not limited in this regard.

[0037] As Figure 4 shown, at block 402, method 400 may include obtaining multiple test traffic for target level testing from multiple initial test traffic, where the target level includes multiple trials. Among them, the test traffic may be a test user or an access request initiated by a test user. For example, the test traffic may be a triggering operation initiated by "Zhang San" on the client (such as the client 102 shown in Figure 1 response to this operation, the client may generate a corresponding click request or activation request as a test traffic. The initial test traffic is generally pre-specified by testers. For example, testers can divide relatively similar test users into a group of test traffic as the initial test traffic according to the similarity of test users. The so-called relatively similar test users may have similar identities, similar locations, or similar usage frequencies of a certain software.

[0038] It is understandable that when testers conduct A / B testing on a certain software, the factors to be considered are relatively diverse. For example, when testers want to test the red envelope distribution function of Alipay and also want to test the layout of the interface elements of Alipay. In order to utilize the functions of different levels of the test application (such as the interface layer, performance layer, security layer, etc.), the testing of the application can be divided into multiple levels based on the test requirements. The test experiments between different levels do not affect each other, and the test traffic is orthogonal in different layers. For example, level 1 can be used to test the layout of the Alipay interface, and level 2 can be used to test the red envelope distribution function of Alipay. In this way, the parallel verification ability of multiple test scenarios can be improved. That is to say, by dividing into multiple levels, the test efficiency can be improved on the premise of ensuring the reliability of the test results.

[0039] In addition, a level can include multiple test experiments. The test experiments within the same level can share the test traffic. For example, multiple experiments within the same level can share 100% of the test traffic according to a preset ratio. For example, experiment 1 in level A can divert 10% of the test traffic, and experiment 2 in level A can divert 10% of the test traffic. This test method similar to pipeline diversion can not only avoid the mutual influence and interference between different experiments, but also improve the test efficiency through parallel testing.

[0040] In some embodiments, after obtaining the initial test traffic, a preset number of traffic can be diverted from the initial test traffic according to a preset ratio as multiple test traffic for testing the target level. It should be noted that one experiment can include multiple test versions and the original version (such as test versions A / B / C and the original version) for a certain function of the application, and the purpose is to test which test version can bring a better user experience to the test users compared to the original version.

[0041] At block 404, method 400 may include allocating a plurality of test traffic to a plurality of trials based on the identification information of the plurality of test traffic, where the identification information is stored in a client and is used to characterize whether the test traffic has participated in a historical test and the test version it has participated in. The test version may be the test version corresponding to software such as an application, a mini-program, a quick application, etc. For example, it may be a version obtained by modifying the current released version according to the modification operations of testers. In some embodiments, the modification operations may be modifying the control color, control shape, etc. of the interface of the application, or may also be the jump method between the pages corresponding to the application. It should be understood that different test versions can be obtained by different ways of modifying the software. In order to determine the user's usage preference for a certain element in the software, the control variable method can be used to modify the same content of the software such as an application in different ways. For example, by modifying the color of the "OK" control of the application to gray, white, etc., two different test versions can be obtained. In other embodiments, different test versions may also be versions obtained by upgrading the software version according to the software update, such as version 1.0, version 1.1, and version 2.0, etc.

[0042] For the test process, generally speaking, in order to ensure the effectiveness of the test, the test cycle may be relatively long or the test front may be stretched. For example, the test cycle can be three days or even longer. Then, in order to ensure that the same test traffic is allocated to the same test version during the test process, it can be determined which test version of which trial the test traffic was allocated to in a historical test trial such as the previous test trial according to the identification information carried by the test traffic. Based on this identification information, the allocation result of the same test traffic can be ensured to be unified. For example, according to the identification information (trial 1, test version A) carried by test traffic A, it can be determined that test traffic A was allocated to test version A of trial 1 in the previous test. On this basis, in the current test, test traffic A can still be allocated to test version A of trial 1. It can be understood that a test cycle may include multiple tests, and different tests have a chronological order. The historical test trial can be any test before the current test.

[0043] It can be understood that the identification information can be held by the client (such as the client shown in Figure 1 ). Generally speaking, the server can allocate the test traffic according to the trial configuration information so that different test traffic is used to test different test versions. After the allocation is completed, the allocation result can be given to the test traffic as a label. From this perspective, the test traffic can carry identification information.

[0044] At block 406, method 400 may include determining corresponding test results by determining feedback of the target test traffic for the target test version in the target experiment based on the allocation result. It can be understood that the test traffic can be allocated to the corresponding test version according to the identification information of the test traffic. Wherein, the feedback may be user behavior data corresponding to the target test traffic or the satisfaction of the user with the test version. The behavior data can be understood as the operation data of the user on the client for the test version, such as clicks, browsing, input, swiping, etc. In some embodiments, the satisfaction of the user with the test version can be determined according to the number of clicks, browsing duration or purchase frequency of the user for the test version, or can be determined according to the questionnaire filled in by the user, which is not limited in this specification. Based on these user behavior data or the satisfaction degree of the user with the test version, the test results of the test version can be determined. For example, the corresponding scoring results can be determined according to the above factors. If the score is relatively high, it indicates that the test version meets the user experience requirements, thus providing a basis for determining the final version of the promoted application.

[0045] As can be seen from the above, the method for testing provided by the embodiments of this specification uses the resources of the client to determine the test versions corresponding to different test traffic. The server only needs to send the corresponding test version or the page data corresponding to the test version to the client, which can effectively reduce the data processing volume and data storage volume of the server and reduce the resource consumption of the server. At the same time, by adopting a layered and shunted method for the test experiment, it can ensure that the experiments between different layers do not interfere with each other, and the experiments in the same layer share the test traffic, thereby improving the test efficiency and reducing the test cost. In addition, using the identification information of the traffic to determine the current test traffic allocation result can ensure that the same traffic, that is, the same client, will not participate in different test versions repeatedly to produce different results, thus ensuring the accuracy and reliability of the test.

[0046] As described above, in the case where the identification information carried by the test traffic to be allocated indicates both the test label number participated in and the test version label number participated in, the test traffic can be simply and directly allocated to the test version in the original experiment in the current test experiment and is also used to test the test version. Then obviously, this allocation method is more logical, scientific and reasonable, and can provide a reliable allocation method for a long test cycle.

[0047] Figure 5 Shows a schematic diagram of allocating test traffic in some embodiments of this specification. As Figure 5As shown, Test 1 may include three test versions and one original version. The labels of the three test versions are Test Version A, Test Version B, and Test Version C respectively. After obtaining Test Traffic 1, the identification information carried by it can be determined. This identification information can specifically be understood as being used to indicate that in the previous test, it was assigned to a certain test version of a certain experiment (for example, in the previous test, this Test Traffic 1 was assigned to Test Version A of Experiment 3). According to this identification information, this Test Traffic 1 can be similarly assigned to Test Version A of Experiment 3 for Test Version A.

[0048] When the server allocates test traffic, it may not allocate all the traffic to specific test versions, or the allocated traffic may not participate in the specific test process. In this case, the identification information carried by the test traffic to be allocated may only indicate the experiment label it participated in. For example, the identification information carried by this test traffic is {Experiment 2, no test version}. That is to say, the identification information that includes the experiment label participated in and does not include any test version that did not participate in this experiment can be understood as that although this test traffic was allocated to Experiment 1 in the previous test, it did not participate in the test of any version in Experiment 1. In this way, Experiment 1 can be randomly re-allocated. However, when re-allocating Experiment 1, the traffic range included in Experiment 1 should be excluded to prevent this test traffic from being allocated to Experiment 1 again.

[0049] Then, to achieve this effect, taking the example of using random numbers for random allocation, an upper limit can be set for the generation of random numbers before generating the random numbers. This upper limit can simply be understood as the value obtained by subtracting the amount of traffic that should be diverted to Experiment 1 from the amount of traffic allocated to the target level. Specifically, according to the proportion of different test versions included in different experiments and the proportion of this experiment at the target level, this test traffic can be randomly allocated. Among them, there are various ways of random allocation. For example, random allocation can be carried out using a probability estimation model, or algorithms such as a random seed number algorithm and a random number algorithm can be used for random allocation.

[0050] Figure 6A schematic diagram showing the distribution of test traffic for other embodiments of this specification is shown. It can be understood that one level includes multiple tests, and there is a sequence among multiple tests when distributing traffic. For example, Test 1 diverts traffic before Test 2, and Test 2 diverts traffic before Test 3. The traffic diversion ratio of a test at one level is determined, and the traffic diversion ratio of a test version in one test is also determined. For example, the test traffic of one level can be 100% traffic. Test 1 diverts 10% of the test traffic, and each of the four versions in Test 1 diverts 2.5% of the traffic. Dividing according to the numbers from the largest to the smallest traffic, then the traffic range of the original version is 0 - 2.5, the traffic range of Test Version A is 2.5 - 5, the traffic range of Test Version B is 5 - 7.5, and the traffic range of Test Version C is 7.5 - 10. For the convenience of division, all data can be increased by one order of magnitude. Then the test traffic of this level is 1000, the traffic of Test 1 is 100, the traffic range of the original version is 0 - 25, the traffic range of Test Version A is 25 - 50, the traffic range of Test Version B is 50 - 75, and the traffic range of Test Version C is 75 - 100.

[0051] As Figure 6 shown, the identification information carried by Traffic A is {Test 1, no test version}, that is, this test traffic was assigned to Test 1 in the previous test but did not participate in the test of any test version of Test 1. As mentioned above, it is necessary to exclude Test 1 to prevent Traffic A from being assigned to Test 1 and avoid the problem of having different distribution results from the previous test. In this way, the upper limit limit of the random number can be: limit = 1000 - 100 = 900. Here, "100" refers to the traffic diverted to Test 1. In this way, the range of the random number can be "0 - 900".

[0052] Due to the exclusion of the participation of Test 1, for the accuracy and convenience of random distribution, the traffic ranges of the test versions of other tests after Test 1 can be relabeled. Taking Test 2 diverting 12% of the traffic, and each of the four versions (original version / Test Version A / Test Version B / Test Version C) of Test 2 diverting 3% of the traffic, Test 3 diverting 12% of the traffic, and each of the four versions (original version / Test Version A / Test Version B / Test Version C) of Test 2 diverting 3% of the traffic as an example, the traffic of Test 2 is 120, the traffic range of the original version is 0 - 30, the traffic range of Test Version A is 30 - 60, the traffic range of Test Version B is 60 - 90, and the traffic range of Test Version C is 90 - 120.

[0053] It can be understood that if the generated random number random is within 0 - 120, then traffic A can be allocated to Experiment 2 and allocated to the corresponding test version according to the range where the specific value is located. Taking the random number 35 as an example, traffic A can be allocated to Test Version A of Experiment 2. In this way, in order to ensure that the subsequent tests in the test cycle are consistent with the allocation result of this test, the identification information corresponding to this allocation result can be given to traffic A to replace the original identification information of traffic A. Then the updated identification information of traffic A is {Experiment 2, Test Version A}.

[0054] If the generated random number is not within 0 - 120, it can be determined that this traffic is not allocated to Experiment 2. To provide an allocation basis for subsequent tests, the result of not being allocated to Experiment 2 can be given to traffic A as a label. To determine which experiment and which test version traffic A is finally allocated to, the allocation result of traffic A can be determined by decreasing the random number. In this way, it is not necessary to re - allocate but continue the subsequent allocation process according to the previous allocation result, thus saving processing resources and improving the allocation efficiency.

[0055] For example, the traffic quantity of Experiment 2 can be subtracted from the generated random number. That is to say, the random number random can be changed to random’ = random - 120. After determining random’, the above - mentioned allocation logic can be repeated until it is determined that traffic A is finally allocated to Test Version n of Experiment m at this level. For example, it is allocated to Test Version C of Experiment 3. Similarly, after determining the final allocation result, the identification information corresponding to this allocation result can be given to traffic A.

[0056] In other embodiments, other methods can also be used to determine the allocation result of traffic A. For example, after excluding the participation of Experiment 1, a random value can be generated through a random number generation algorithm to determine which traffic range the random value falls into. For example, when the random value is 200, it can be determined that it falls into Test Version B of Experiment 3 (the traffic range of Experiment 3 is 120 - 240, and the traffic range of Test Version B is 180 - 210). In this way, the information that traffic A is not allocated to Experiment 1, not allocated to Experiment 2, and is allocated to Test Version B of Experiment 3 can be given to traffic A as the new identification information of traffic A.

[0057] In addition, when the server allocates test traffic, it may not allocate all the traffic, or may not have allocated the newly arrived traffic yet. The identification information carried by such traffic may not participate in any experiment and has not been assigned to any test version. That is to say, this traffic did not participate in the test and was not assigned to a specific test version in the previous test of the test cycle. Then, the traffic can be randomly allocated to determine the allocation result of the traffic in the current test. For example, a random value can be obtained by means of a random number, and the allocation result of the traffic can be determined based on the range in which the random number is located.

[0058] Figure 7 shows a schematic diagram of the process of test traffic allocation provided by some embodiments of this specification. As Figure 7 shown, the identification information carried by traffic A does not include the experiment participation mark or the test version mark participated in, that is, the identification information can be {no experiment, no test version}. It can be understood that to determine the allocation result of traffic A, a similar method as described above can be used to sequentially judge whether traffic A falls into a certain test version of a certain experiment according to the order of the experiments.

[0059] As Figure 7 shown, a random number can be generated, and the range of this random number is 0 - 1000. If the random number falls within the traffic range of Experiment 1 and within the traffic range of Test Version A of Experiment 1. Then traffic A can be allocated to Test Version A of Experiment 1, and this allocation result can be used as the identification information to be given to traffic A. If the random number does not fall within the traffic range of Experiment 1, then the traffic of Experiment 1 can be excluded based on the random number, and a new random number can be determined. If the newly determined random number falls within the traffic range of Experiment 2 and within Test Version B of Experiment 2, then traffic A can be allocated to Test Version B of Experiment 2, and this allocation result can be used as the identification information to be given to traffic A. This identification information can be (not allocated to Experiment 1, allocated to Test Version B of Experiment 2). If the random number does not fall within the traffic range of Experiment 2, then the traffic of Experiment 2 can be excluded based on the random number, and a new random number can be determined. Repeat the above steps until the allocation result of traffic A is determined.

[0060] Similarly, in other embodiments, the traffic ranges of the test versions included in multiple tests can be uniformly numbered. For example, the traffic range of Test 1 is 0 - 100 (the traffic range of the original version is 0 - 25, the traffic range of Test Version A is 25 - 50, the traffic range of Test Version B is 50 - 75, and the traffic range of Test Version C is 75 - 100), and the traffic range of Test 2 is 100 - 220 (the traffic range of the original version is 100 - 130, the traffic range of Test Version A is 130 - 160, the traffic range of Test Version B is 160 - 190, and the traffic range of Test Version C is 190 - 220). On this basis, the allocation result of the test traffic can be determined by randomly generating a random number within the range of 0 - 1000. For example, when the random number is 170, the test traffic can be allocated to Test Version B of Test 2. In this way, the identification information of the test traffic is not allocated to Test 1 and is allocated to Test Version B of Test 2.

[0061] It can be understood that a test cycle can include n tests. To more comprehensively count the test results, the final test result of the test cycle can be determined according to the statistical values of the n feedback results corresponding to the n tests. Taking the feedback result of the user's stay duration on the page of the application as an example to illustrate the process of determining the test result. For example, in the first test, the stay duration of User A on the page of Test Version A is 2s, the stay duration of User B on the page of Test Version B is 1s, and the stay duration of User D on the page of Test Version C is 4s; in the second test, the stay duration of User A on the page of Test Version A is 3s, the stay duration of User B on the page of Test Version B is 2s, and the stay duration of User D on the page of Test Version C is 6s; in the third test, the stay duration of User A on the page of Test Version A is 1s, the stay duration of User B on the page of Test Version B is 2s, and the stay duration of User D on the page of Test Version C is 5s. In this way, within the entire test cycle, the stay duration of User A on the page of Test Version A is 2s + 3s + 1s = 6s, the stay duration of User B on the page of Test Version B is 1s + 2s + 2s = 5s, and the stay duration of User D on the page of Test Version C is 4s + 6s + 5s = 15s. That is to say, Test Version C is more popular among users, and users are more inclined to use Test Version C. Then, Test Version C can be used as the final version of the application.

[0062] In practical applications, users of different ages have different usage habits of Taobao, and users in different regions also have different usage habits of Taobao. For example, young people may be more inclined to buy fashionable and trendy goods, while middle-aged and elderly people may pay more attention to the practicality and cost-effectiveness of goods. Based on this, users can be grouped according to their information or characteristics. Different groups of users can be used to test different functions or different applications. Among them, the information of users can include the identity information of users and the location information of users. Among them, the identity information of users can include the ID card information of users or other identification information that can uniquely identify the identity of the user, the mobile phone number of users or other identity information collected with the authorization of users. The location information of users can be the geographical location where the users are located, such as the city, region where the users are located, or the longitude and latitude information of the location where they are located.

[0063] It can be understood that users in the same group can be users in the same location area. The same location area here does not require exactly the same location in terms of longitude and latitude, but can be a preset certain range. By presetting a reasonable range, users who may have a similar relationship can be screened out from all parties.

[0064] It can be understood that during the software development process, the test team usually formulates a test plan according to the requirements of the project, the development progress and resource conditions, and sets corresponding test requirements, test duration and test end conditions. Among them, the test end condition refers to the condition or standard for the end of the test work. If the test requirements are different or the functions or pages to be tested are different, the test duration required or the test end conditions are different. On this basis, users can set corresponding test plans according to the test requirements or the objects to be tested. For example, for a relatively simple test, a shorter test duration can be set; while for a relatively complex test, a longer test duration can be set. In this way, by setting a reasonable test plan, the effectiveness and efficiency of the test work can be improved.

[0065] Figure 8 A block diagram of a computing device 800 suitable for use in implementing the embodiments of the present invention is schematically shown. The computing device 800 may be a device for implementing the Figure 4 method 400 shown. As Figure 8As shown, computing device 800 includes a processing unit (CPU) 801 that can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or computer program instructions loaded from storage unit 808 into random access memory (RAM) 803. In RAM 803, various programs and data required for the operation of computing device 800 can also be stored. CPU 801, ROM 802, and RAM 803 are connected to each other via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0066] A plurality of components in computing device 800 are connected to I / O interface 805, including: input unit 806, output unit 807, storage unit 808. Processing unit 801 executes the various methods and processes described above, such as executing method 400. For example, in some embodiments, the various processes or operations described above can be implemented as a computer software program, which is stored in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto computing device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, the various methods and processes described above can be executed, such as executing one or more operations of method 400. Alternatively, in other embodiments, CPU 801 can be configured to execute the various methods and processes described above, such as executing one or more actions of method 400, by any other suitable means (e.g., by means of firmware).

[0067] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0068] The computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present invention.

[0069] These computer - readable program instructions can be provided to the processing unit of a processor, a general - purpose computer, a special - purpose computer, or other programmable data - processing device in a voice - interaction device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data - processing device, a device is produced that implements the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, and these instructions cause the computer, programmable data - processing device, and / or other devices to work in a specific manner.

[0070] The above - described specific embodiments of the present specification have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be executed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0071] The embodiments of the present invention have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

[0072] The above are only optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for testing, comprising: Acquire a plurality of test flows for target level testing from a plurality of initial test flows, the target level including a plurality of experiments; Based on the identification information of the multiple test flows, the multiple test flows are allocated to the multiple tests, the identification information is stored in the client and is used to characterize whether the test flows have participated in historical tests and the test versions participated in; as well as Based on the allocation result, the corresponding test result is determined by determining the feedback of the target test traffic for the target test version in the target experiment.

2. The method according to claim 1, wherein based on the identification information of the test traffic, allocating the test traffic to the plurality of tests comprises: Based on the identification information of the first test traffic, determining that the first test traffic is allocated to the first test version in the first trial in the historical test; as well as The first test traffic is allocated to a first test version of a first trial in the target tier.

3. The method according to claim 1, wherein based on the identification information of the test traffic, allocating the test traffic to the plurality of tests comprises: Based on the identification information of the second test traffic, determining that the second test traffic is allocated to the second test in the historical test but is not allocated to any second test version of the second test; generating a random number based on the number of the plurality of test flows by a random number function; as well as Based on the random number, the second test traffic is allocated to the target test version of the target test, and the target test is any test after the second test in the target level. The distribution of the random number in the multiple test traffic corresponds to the diversion ratio of the target test in the target level.

4. The method according to claim 3, further comprising: Based on the allocation result of the second test traffic, the identification information of the second test traffic is updated, and the identification information indicates that the second test traffic is allocated to the target test version in the second test in the target level test test.

5. The method according to claim 3, wherein generating a random number based on the number of the plurality of test flows by a random number function comprises: determining a difference between the number of the plurality of test flow rates and the number of test flow rates corresponding to the second test; Generate a corresponding random number based on the difference through a random number function.

6. The method according to claim 1, wherein allocating the test traffic to the plurality of tests based on the identification information of the test traffic comprises: Based on the identification information of the third test traffic, determining that the third test traffic is not allocated to any test version in the historical test in the historical test; generating a random number based on the number of the plurality of test flows by a random number function; Based on the random number, determining whether the third test traffic is within any test version of the multiple tests; as well as In response to the third test traffic not being in any of the test versions, determining that the third test traffic is in a target test version of a target test among the multiple tests based on the random number and the number of test traffic corresponding to a test in the target level.

7. The method according to claim 1, further comprising: Allocating, by a server connected to the client, a preset number of test flows of the initial test flow to a first test of the target level, wherein the first test includes a plurality of test versions; Evenly distribute the preset number of test flows to the multiple test versions; as well as Based on the result of the allocation, identification information corresponding to each of the preset number of test flows is determined.

8. The method according to claim 1, obtaining a plurality of test flows for target layer testing from a plurality of initial test flows comprises: Acquire test configuration information, wherein the test configuration information includes a plurality of test traffic diversion strategies; as well as Based on the diversion strategy, multiple test flows for target level testing are obtained from the multiple initial test flows.

9. The method according to claim 8, wherein the diversion strategy comprises at least one or more of the following: the ratio of the number of flows allocated to the first level among the plurality of test flows and the ratio of the number of flows allocated to the second level among the plurality of test flows; a ratio of the number of flows of the first test allocated to the first level to the number of flows of the first level; A ratio of the amount of traffic of the first test version allocated to the first test to the amount of traffic of the first test.

10. The method according to claim 1, further comprising: Determine a plurality of feedbacks corresponding to the target test flow in a plurality of tests, wherein the plurality of tests include a target test, a test trial before the target test, and a test trial after the target test; as well as Based on the statistical values ​​of the multiple feedbacks, corresponding test results are generated.

11. A device for testing, comprising: A traffic acquisition module, configured to acquire a plurality of test traffics for a target level test from a plurality of initial test traffics, wherein the target level includes a plurality of tests; A traffic distribution module is configured to distribute the multiple test flows to the multiple tests based on identification information of the multiple test flows, the identification information being stored in the client and used to characterize whether the test flows have participated in historical tests and the test versions they have participated in; as well as The test result determination unit is configured to determine the corresponding test result by determining the feedback of the target test traffic for the target test version in the target test based on the allocation result.

12. A processor, configured to execute the method according to any one of claims 1 to 10.

13. A computing device comprising: Memory; as well as A processor according to claim 12.

Images