A method, apparatus and equipment for A / B testing

Abstract

This disclosure provides a method, apparatus, device, and storage medium for A / B testing, relating to the field of computer technology, and particularly to the field of computer testing technology. The specific implementation scheme is as follows: Traffic is randomly divided into group A and group B, with group A and group B having equal amounts of traffic; group A is randomly and equally divided into at least two subgroups of traffic; group B is also randomly and equally divided into at least two subgroups of traffic, with each subgroup of group B having equal amounts of traffic to each subgroup of group A; the subgroups of traffic are then permuted and combined to obtain at least four pairs of subgroups; multiple comparative tests are performed using at least four pairs of subgroups of traffic to obtain the A / B test results. The A / B test results obtained using the above method are more stable and reliable compared to the test results obtained using only one pair of traffic groups.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the field of computer testing technology. Background Technology

[0002] A / B testing is a primary tool for internet companies to analyze experimental data, refine operations, and promote product strategies. It involves randomly sampling traffic into two groups, control group A and experimental group B, conducting hypothesis testing on each group, and then evaluating the differences in indicators between the two groups. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for A / B testing.

[0004] According to one aspect of this disclosure, an A / B testing method is provided, comprising: randomly dividing traffic into group A traffic and group B traffic, wherein group A traffic and group B traffic are of equal quantity; randomly dividing group A traffic into at least two subgroup traffic; randomly dividing group B traffic into at least two subgroup traffic, wherein each subgroup traffic of group B traffic is equal to each subgroup traffic of group A traffic; permuting and combining the subgroup traffic to obtain at least four pairs of subgroup traffic; and using at least four pairs of subgroup traffic, performing multiple comparative tests to obtain A / B test results.

[0005] According to another aspect of this disclosure, an A / B testing apparatus is provided, comprising: a flow group division module for randomly dividing flow into group A flow and group B flow, wherein the flow of group A flow is equal to the flow of group B flow; a group A flow division module for randomly dividing group A flow into at least two subgroup flow; a group B flow division module for randomly dividing group B flow into at least two subgroup flow, wherein each subgroup flow of group B flow is equal to each subgroup flow of group A flow; a subgroup flow pairing module for arranging and combining the subgroup flow to obtain at least four pairs of subgroup flow; and an A / B testing module for performing multiple comparative tests using at least four pairs of subgroup flow to obtain A / B test results.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described AB test method.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described AB test method.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described AB test method.

[0009] The A / B testing method, apparatus, equipment, and storage medium disclosed herein subdivide group A and group B traffic into multiple subgroup traffic, cross-pair these subgroup traffic to obtain multiple pairs of subgroup traffic, and perform multiple comparative tests using these multiple pairs of subgroup traffic to obtain A / B test results. Compared with the test results obtained by using a single pair of traffic for A / B testing, the test results obtained by using the above method are more stable and reliable.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This is a flowchart illustrating the method for implementing A / B testing according to the first embodiment of this disclosure;

[0013] Figure 2 This is a schematic diagram illustrating the grouping of traffic according to the second embodiment of this disclosure;

[0014] Figure 3 This is a flowchart illustrating the method for implementing A / B testing according to the second embodiment of this disclosure;

[0015] Figure 4 This is a flowchart illustrating the method for implementing A / B testing according to the third embodiment of this disclosure;

[0016] Figure 5 This is a schematic diagram of the process for implementing P-value correction according to the fourth embodiment of this disclosure;

[0017] Figure 6 This is a schematic diagram of the structure of the AB test apparatus according to an embodiment of this disclosure;

[0018] Figure 7 This is a block diagram of an electronic device used to implement the AB testing method of the embodiments of this disclosure. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] In this disclosure, A / B testing, also known as A / B experiment, refers to the process of taking a portion of traffic (users) and randomly dividing them into two groups. One group maintains the existing strategy and is called the control group A, while the other group uses an improved strategy and is called the experimental group B. Statistical methods are used to analyze the differences in indicators between the two groups to evaluate whether they meet expectations.

[0021] Figure 1 The flowchart of an embodiment of the present disclosure for implementing A / B testing is shown, as follows: Figure 1 As shown, it includes:

[0022] Operation S110 randomly divides the traffic into group A traffic and group B traffic, with group A traffic and group B traffic being equal in quantity;

[0023] For online products, traffic mainly refers to users accessing the product within a certain period of time; for algorithm or data experiments, traffic mainly refers to the input data used to execute algorithms or perform data calculations.

[0024] Group A flow rate is the flow rate used in the control group of the AB test, and Group B flow rate is the flow rate used in the experimental group test.

[0025] Random division is used when dividing traffic to ensure that the traffic in group A and group B is homogeneous, that is, the traffic in group A and group B has similar distribution or characteristics.

[0026] Only by ensuring that the flow rates of Group A and Group B are equal and homogeneous can the test results of the control group be comparable to those of the reference group, and only then can the results of the AB test be valid.

[0027] Operation S120: Randomly and equally divide the traffic of group A into at least two subgroups of traffic;

[0028] In this case, each of the at least two subgroups of traffic in group A is obtained by randomly dividing the traffic in group A, and each subgroup has an equal amount of traffic.

[0029] Group A traffic can be randomly divided into at least two subgroups, or into three or more subgroups. The more subgroups, the higher the stability and reliability of the A / B test; however, the number of subsequent hypothesis tests and data calculations will increase exponentially. Implementers can flexibly determine the appropriate number based on specific implementation needs, conditions, and desired results.

[0030] Operation S130: Randomly and equally divide the traffic of group B into at least two subgroups of traffic, with each subgroup of traffic of group B having the same amount of traffic as each subgroup of traffic of group A.

[0031] In this case, each of the at least two subgroups of traffic in group B is obtained by randomly dividing the traffic in group B, and each subgroup has an equal amount of traffic.

[0032] The number of subgroups of traffic in group B is the same as the number of subgroups of traffic in group A; the amount of traffic in each subgroup of traffic in group B is also equal to the amount of traffic in each subgroup of traffic in group A.

[0033] In this way, each subgroup of the B group flow can be combined with each subgroup of the A group flow to form multiple pairs of subgroup flows that can be used for control group testing and experimental group testing respectively.

[0034] Operation S140: Arrange and combine the subgroup traffic to obtain at least four pairs of subgroup traffic;

[0035] Since the flow rates of group A and group B are equal and homogeneous; the number of subgroups into which group B is divided is the same as the number of subgroups into which group A is divided; and the flow rates of each subgroup are equal; therefore, the number of subgroups into which group B is divided is also equal and homogeneous with that of group A. Any two subgroups can be a pair of subgroups, which can be used for control group testing and experimental group testing, respectively.

[0036] Since the traffic in group A is divided into at least two subgroups, and the traffic in group B is also divided into at least two subgroups, for a total of four subgroups, by arranging and combining each pair of subgroups in the four subgroups, at least four pairs of subgroups can be obtained.

[0037] For example, if we divide the traffic of group A into the first subgroup traffic and the second subgroup traffic, and further divide the traffic of group A into the third subgroup traffic and the fourth subgroup traffic, then we can obtain at least the following four pairs of subgroup traffic:

[0038] First subgroup flow and third subgroup flow;

[0039] The flow rates of the first and fourth subgroups;

[0040] Second subgroup flow and third subgroup flow;

[0041] The flow rates of the second and fourth subgroups.

[0042] Operate S150, use at least four pairs of subgroup traffic, and conduct multiple comparative tests to obtain AB test results.

[0043] In practical applications, each pair of subgroup traffic can be used for a comparative test to obtain multiple comparative test results, and then the A / B test results can be determined based on the multiple comparative test results.

[0044] When determining the A / B test results based on multiple comparative test results, corresponding strategies or rules can be set according to implementation needs and implementation effects. For example, if any test result shows "significant" improvement, the A / B test result can be set as "significant" improvement; or, if more than half of the test results show "significant" improvement, the A / B test result can be set as "significant" improvement.

[0045] In practical applications, A / B testing is usually implemented through hypothesis and hypothesis testing. Therefore, the multiple comparison tests here are actually multiple hypothesis tests based on a hypothesis test for each pair of subgroup traffic. The A / B test results can then be determined based on the results of the multiple tests.

[0046] Multiple hypothesis testing refers to treating multiple single hypothesis tests as a whole, also known as a family of tests, and then testing each hypothesis in this family of tests simultaneously to obtain a final test result.

[0047] In this embodiment, the traffic of group A and group B is split twice. Multiple pairs of subgroup traffic are formed by cross-pairing the subgroup traffic and multiple comparison tests (multiple hypothesis testing) are performed. There are multiple test results for reference. The AB test results obtained in this way are more stable and reliable than the AB test results obtained by using only one pair of traffic: group A traffic and group B traffic, which greatly reduces the randomness and accidental nature of AB test.

[0048] Figure 2 This embodiment illustrates a specific implementation of traffic grouping according to another embodiment of the present disclosure. In this embodiment, a sampling platform 201 performs a "sampling initiation" operation, and then sends the operation request to a sampling engine 202. Upon receiving the "sampling initiation" operation request, the sampling engine 202 performs data sampling (i.e., traffic partitioning) and returns the sampling results—group A data and group B data—to the sampling platform 201. After obtaining the sampling results, the sampling platform 201 displays the results to the user who initiated the sampling.

[0049] exist Figure 2In the embodiments of this disclosure, when performing data sampling, the sampling engine 202 will: on the one hand, randomly extract 10% of the traffic as group A data; on the other hand, randomly extract 10% of the traffic as group B data; then, randomly extract 5% of the data from group A as subgroup A1 data, and the remaining 5% as subgroup A2 data; and randomly extract 5% of the data from group B as subgroup B1 data, and the remaining 5% as subgroup B2 data.

[0050] In this context, the traffic of subgroup A1 and subgroup A2 are each subsets of the traffic of group A, and there is no overlap between the traffic of subgroup A1 and subgroup A2. Similarly, the traffic of subgroup B1 and subgroup B2 are each subsets of the traffic of group B, and there is no overlap between the traffic of subgroup B1 and subgroup B2.

[0051] The mapping relationship between the above data and groups (for example, data 1 belongs to group A1 in group A) will be written into the database for subsequent indicator data calculation.

[0052] exist Figure 2 In the illustrated embodiment, dividing group A data into subgroups A1 and A2, and dividing group B data into subgroups B1 and B2, are all performed in the sampling engine 202 and do not affect the operation and presentation of the sampling platform 201. Thus, the AB testing method of this disclosure can be implemented simply by adding a secondary sampling operation for group A and group B data to the sampling engine 202, without modifying the application layer's processes or operations. This allows for improvement of AB test results at a relatively low cost.

[0053] Figure 3 This disclosure illustrates Figure 2 The flowchart of the method for implementing A / B testing in the illustrated embodiment is as follows: Figure 3 As shown, it includes:

[0054] Operation S310 randomly divides the data into group A and group B.

[0055] Operation S320 randomly and equally divides the data in group A into two subgroups: subgroup A1 and subgroup A2.

[0056] Operation S330 randomly and equally divides the data in group B into two subgroups: subgroup B1 and subgroup B2.

[0057] For details regarding how data is partitioned in operations S310 to S330, please refer to [link / reference]. Figure 2 The relevant descriptions will not be repeated here.

[0058] Operation S340 arranges and combines the subgroup data to obtain six pairs of subgroup data: A1A2, B1B2, A1B1, A1B2, A2B1 and A2B2;

[0059] In this embodiment of the disclosure, when arranging and combining subgroup data, in addition to retaining the original AB grouping to ensure that the data of group A is used as much as possible for the control group and the data of group B is used as much as possible for the experimental group, two additional pairs of subgroup data are added: A1A2 and B1B2. On the one hand, the data of subgroup A2 can be used as a supplement to the data of group B, and the data of subgroup B1 can be used as a supplement to the data of group A, increasing the testing scenarios; on the other hand, the homogeneity between A1 and A2, as well as between B1 and B2, can be detected through the above-mentioned pairing test, thereby ensuring that the experiment is effective and the test results are reliable.

[0060] Operate S350, use six pairs of subgroup data, perform multiple hypothesis testing, and obtain AB test results;

[0061] In this embodiment of the disclosure, the comparative test is performed by hypothesis testing. The AB test result is obtained by performing multiple comparative tests based on multiple subgroup data.

[0062] Operate the S360 to test the validity of the six pairs of subgroup data;

[0063] In this embodiment of the disclosure, after obtaining the AB test results, the effectiveness of the test will be further checked.

[0064] Among them, the validity test mainly proves that the data used in the test is valid, the strategy is appropriate, and the operation is error-free.

[0065] Specifically, the following effectiveness tests were performed in the embodiments of this disclosure:

[0066] 1) Homogeneity test;

[0067] The homogeneity test is mainly to ensure that the secondary partitioning (e.g., dividing group A data into subgroups A1 and A2; dividing group B data into subgroups B1 and B2) is homogeneous (whether the flow partitioning is random). If they are not homogeneous, the test results obtained from each pair of subgroup flows are not comparable, and the AB test is invalid.

[0068] Specifically, in this embodiment of the disclosure, the core indicators of the traffic of the two pairs of subgroups A1A2 and B1B2 are tested. If the core indicators obtained by comparing A1A2 or B1B2 are unevenly distributed with outliers, or if there are large differences in the indicators between different categories, the experiment can be determined to be invalid.

[0069] Therefore, in order to perform a homogeneity test, this embodiment of the disclosure proposes the following hypotheses in accordance with hypothesis testing methods:

[0070] H0: The experiment is valid (satisfies the homogeneity test).

[0071] H1: The experiment is invalid (it does not meet the homogeneity test).

[0072] This hypothesis is a double test for homogeneity. By obtaining core indicator data, the test is performed separately for the control group A1A2 and the experimental group B1B2. If both tests fail, the splits are considered homogeneous, H0 cannot be rejected, and the experiment is valid. Details are shown in Table 1 below:

[0073] A1A2 Homogeneity Test B1B2 Homogeneity Test Test validity verification pass pass pass pass Not approved Not approved Not approved pass Not approved Not approved Not approved Not approved

[0074] Table 1

[0075] 2) Validation of effectiveness;

[0076] The effectiveness test is mainly to verify the validity of the test results. In this embodiment, it is measured by whether the index values ​​cross. If the core index values ​​show a significant crossover between the experimental group and the control group, for example, if an A / B test is used to verify the effect of the modified strategy, and the effect of experimental group A1 is significantly better than that of control group B1, while the effect of control group B1 is significantly better than that of experimental group A2, then it is impossible to determine whether the effect of the strategy change has improved or worsened, and therefore the experiment is deemed invalid.

[0077] Therefore, in order to verify the effectiveness of the results, this embodiment of the present disclosure proposes the following hypotheses in accordance with the hypothesis testing method:

[0078] H0: The experiment is valid (satisfies the cross-validation).

[0079] H1: The experiment is invalid (cross-validation is not satisfied).

[0080] In this embodiment of the disclosure, the hypothesis is a cross-validation multiple test. After sorting the core indicators of each subgroup data pair by mean, six scenarios can be formed (see Table 2 for details):

[0081]

[0082] Table 2

[0083] Among them, except for AABB and BBAA which do not have any overlap and do not need to be tested, the other four cases require four-fold hypothesis testing on B1A1, B1A2, A1B2, and A2B2. If there are two sets of tests with significant differences in opposite directions in the overlap part, then A and B have significant overlap, H0 is rejected, and the experiment is invalid; otherwise, H0 cannot be rejected, and the experiment is valid.

[0084] Operation S370 determines whether the test validity check has passed. If it has passed, continue to operation S380. If it has failed, return to operation S320, redistribute the data of group A and group B into random equal parts, and perform subsequent tests and test validity checks.

[0085] In this embodiment of the disclosure, if both 1) the homogeneity test and 2) the effectiveness test are passed, the effectiveness test can be determined to be passed; otherwise, it is not passed.

[0086] Operate S380 to perform a test evaluation based on the AB test results and obtain the test evaluation results.

[0087] exist Figure 3 The embodiment of this disclosure shown includes an added test validity check, which further ensures the validity of the test and avoids being misled by invalid test results.

[0088] Figure 4 The flowchart of another embodiment of the present disclosure for implementing A / B testing is shown, as follows: Figure 4 As shown, it includes:

[0089] Operation S4010 randomly divides the data into group A and group B.

[0090] Operation S4020: Randomly and equally divide the data in group A into two subgroups: subgroup A1 and subgroup A2.

[0091] Operation S4030: Randomly and equally divide the data in group B into two subgroups: subgroup B1 and subgroup B2.

[0092] Operation S4040 arranges and combines the subgroup data to obtain six pairs of subgroup data: A1A2, B1B2, A1B1, A1B2, A2B1 and A2B2;

[0093] Operate S4050, use six pairs of subgroup data, perform multiple hypothesis testing, and obtain AB test results;

[0094] Operate S4060 to test the validity of the six pairs of subgroup data;

[0095] Operation S4070: Determine whether the test validity check has passed. If it has, proceed to operation S4080. If it has not, return to operation S4020 and re-randomly divide the data in group A and group B into equal parts.

[0096] Operate S4080 to perform a test evaluation based on the AB test results and obtain the test evaluation results;

[0097] Operate S4090, adjust the P-value based on the test evaluation results, and use the adjusted P-value to verify the validity of the test.

[0098] The P-value represents the probability of a more extreme outcome occurring in the sample observations when the null hypothesis is true. A small P-value indicates a low probability of the hypothetical scenario occurring; if it does occur, according to the principle of low probability, we have reason to reject the null hypothesis. The smaller the P-value, the stronger our reason for rejecting the null hypothesis. In short, a smaller P-value indicates a more significant result.

[0099] Because the testing process in operation S4080 involves multiple hypothesis testing, the probability of a Type I error increases sharply as the number of hypothesis tests increases.

[0100] Therefore, the p-value needs to be adjusted according to the number of hypothesis tests to reduce the probability of Type I error caused solely by multiple hypothesis tests.

[0101] Operate S4100 to output test evaluation results.

[0102] The test evaluation was performed again using the corrected P-value, and the corresponding test evaluation results were output.

[0103] exist Figure 4 In the embodiments of this disclosure shown, a P-value correction process is added to address the problem of increased Type I error probability caused by multiple hypothesis testing. This process eliminates the error problem caused by multiple hypothesis testing, making the A / B test results more accurate and better reflecting the substantial issue of whether the effect of the strategy improvement is significant.

[0104] In other embodiments of this disclosure, a step of correcting the P-value may also be added to operation S4060 to ensure that the validity test involving multiple hypothesis testing is also a corrected and more accurate test.

[0105] Figure 5 The process of implementing the modified P value according to another embodiment of this disclosure is shown, such as Figure 5 As shown in this embodiment, data group A is divided into subgroups A1 and A2, and data group B is divided into subgroups B1 and B2. These are paired into four subgroups for A / B testing: A1B1, A2B1, A1B2, and A2B2. After the tests are completed, the following operations are performed to evaluate the test results:

[0106] Operate S510 to perform multiple hypothesis testing for each indicator;

[0107] In this embodiment of the disclosure, a four-fold hypothesis test is performed for each indicator, and each hypothesis includes:

[0108] H0: There were no significant differences in the indicators between the experimental group and the control group (all AB combinations were not significant);

[0109] H1: There are significant differences in the indicators between the experimental group and the control group (only one of the AB combinations needs to be significant).

[0110] The four hypothesis tests yielded the following results: result 1 for the A1B1 subgroup data, result 2 for the A2B1 subgroup data, result 3 for the A1B2 subgroup data, and result 4 for the A2B2 subgroup data. The p-value for result 1 is p1; the p-value for result 2 is p2; the p-value for result 3 is p3; and the p-value for result 4 is p4.

[0111] In this embodiment of the disclosure, the multiple significance test of the index leads to an increase in the probability of a Type I error to 1 - 0.954 = 0.185.

[0112] Operate S520, correct P value using BH method;

[0113] In this embodiment of the disclosure, the BH method (Benham and Hochberg FDR) is used for p-value adjustment. For m hypotheses, the corresponding p-values ​​can be calculated using the following BH method testing procedure:

[0114] Step 1: Sort the p values ​​of the m hypotheses to obtain p (1) ≤…≤p (m)

[0115] Step 2: Determine the optimal false detection rate (FDR):

[0116] Where, p (j) It is the p-value of the j-th hypothesis test, α (j) It is the significance level given for the j-th hypothesis.

[0117] Step 3: If p (j) ≤p (k) Then reject H0. (j)

[0118] The above method can control the probability of a Type I error to be less than the given significance level α. Furthermore, according to the principle of the BH method, if the given significance level α is 0.05, the adjusted P-value P' can be directly obtained from...

[0119] Calculated, where r represents the rank corresponding to the p-value of one of the test results, if p′ exists. (r) If the value is less than 0.05, then the null hypothesis H0 is rejected at a significance level of 5%.

[0120] After the above p-value correction process, the corrected p-values ​​are obtained: p1', p2', p3', and p4'.

[0121] The significant multiple test can then be re-evaluated using the corrected p-value to obtain the A / B test results. In this embodiment of the disclosure, if any one of the four hypothesis tests results in a "significant" improvement in effect, the A / B test result is determined to be a "significant" improvement in effect.

[0122] Operate S530 to output AB test results.

[0123] In this embodiment of the disclosure, by using the above-described BH method to correct the P-value, the probability of the first type of error can be adjusted back to 0.05, thereby making the AB test results more accurate.

[0124] It should be noted that Figures 1 to 5 The embodiments shown are merely illustrative examples of the A / B testing method disclosed herein and are not intended to limit the implementation methods or application scenarios.

[0125] In practical applications, implementers can also apply the AB testing method of this disclosure to an AB testing platform, providing an option to set the subgroup traffic segmentation option. If the subgroup traffic segmentation option is selected, the AB testing method of this disclosure is executed. For example, the traffic of group A can be further randomly divided into at least one subgroup traffic. When a user enters the AB testing platform and views the experimental report, the platform will obtain the corresponding AB testing information and determine whether the AB test has segmented the traffic. If so, a subgroup traffic segmentation indicator will be displayed on the page, along with the AB test report results. Specifically, the AB test report results with segmented traffic can display: the test validity verification results, including homogeneity verification and effect validity verification; if the test is valid, all data corrected by the BH method and the AB test results will be displayed; if the test is invalid, no data or AB test results will be displayed, only the conclusion that the test is invalid will be given.

[0126] In this way, it is compatible with both traditional AB testing without subgroup traffic partitioning and the AB testing method of this disclosure that further partitions traffic into subgroups, allowing users to flexibly choose the appropriate AB testing method according to their needs.

[0127] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0128] According to embodiments of this disclosure, this disclosure also provides an AB testing apparatus, such as... Figure 6As shown, the device 60 includes: a flow group division module 601, used to randomly divide the flow into group A flow and group B flow, where group A flow and group B flow are equal; a group A flow division module 602, used to randomly divide group A flow into at least two subgroup flow; a group B flow division module 603, used to randomly divide group B flow into at least two subgroup flow, where each subgroup flow of group B flow is equal to each subgroup flow of group A flow; a subgroup flow pairing module 604, used to arrange and combine the subgroup flow to obtain at least four pairs of subgroup flow; and an AB test module 605, used to perform multiple comparative tests using at least four pairs of subgroup flow to obtain AB test results.

[0129] According to one embodiment of this disclosure, the device 60 further includes: a test validity verification module for verifying the test validity of at least four pairs of subgroup traffic.

[0130] According to one embodiment of this disclosure, the test validity verification module is specifically used to perform homogeneity verification on at least four pairs of subgroup traffic.

[0131] According to one embodiment of this disclosure, the test validity verification module is specifically used to verify the effectiveness of at least four pairs of subgroup traffic.

[0132] According to one embodiment of this disclosure, the device 60 further includes: a test evaluation module, used to perform test evaluation based on AB test results to obtain test evaluation results; and a P-value correction module, used to correct the P-value based on the test evaluation results.

[0133] According to one embodiment of this disclosure, the P-value correction module includes: an optimal false detection rate determination submodule, used to determine the optimal false detection rate (FDR) based on the test evaluation results; and a P-value correction submodule, used to correct the P-value based on the FDR.

[0134] According to one embodiment of this disclosure, the device 60 further includes: an option setting module, used to set subgroup traffic options; if the subgroup traffic option is selected, the AB test method of this disclosure described above is executed.

[0135] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0136] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0137] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0138] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0139] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the A / B testing methods of this disclosure. For example, in some embodiments, the A / B testing methods of this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the A / B testing methods of this disclosure described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the A / B testing methods of this disclosure by any other suitable means (e.g., by means of firmware).

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0145] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0146] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for A / B testing, comprising: The traffic is randomly divided into group A and group B, with the traffic in group A being equal in amount to the traffic in group B. The traffic in group A is randomly and equally divided into at least two subgroups of traffic; The traffic in group B is randomly and equally divided into at least two subgroups of traffic, and each subgroup of traffic in group B is equal to the traffic in each subgroup of traffic in group A. The subgroup flows are arranged and combined to obtain at least four pairs of subgroup flows; Using the at least four pairs of subgroup traffic, perform multiple comparative tests to obtain AB test results; The validity of the at least four pairs of subgroup flow rates was tested. During the validity test of the at least four pairs of subgroup traffic, multiple hypothesis tests are performed to obtain the P-value corresponding to each pair of subgroup traffic. Determine the optimal false detection rate (FDR); Based on the FDR, correct the P value corresponding to the flow of each pair of subgroups; The validity test of the four pairs of subgroup traffic includes: The effectiveness of the at least four pairs of subgroup flow rates was tested. The effectiveness test involves sorting the core indicator values ​​of each subgroup of traffic in group A and each subgroup of traffic in group B according to their mean. If there is an overlap between the sorting results of group A and group B, then at least four pairs of subgroup traffic are subjected to hypothesis testing. If at least two pairs of subgroup traffic have significant differences in their tests, and the directions are opposite, then the core indicator values ​​of group A and group B have significantly overlapped, and the effectiveness test fails; otherwise, the effectiveness test passes.

2. The method according to claim 1, wherein, The validity test of the four pairs of subgroup traffic includes: Homogeneity test is performed on the flow rates of the at least four pairs of subgroups.

3. The method according to claim 1, further comprising: If the option to divide traffic into subgroups is selected, the method described in claim 1 is executed.

4. An apparatus for AB testing, comprising: The traffic group division module is used to randomly divide traffic into group A traffic and group B traffic, wherein the amount of group A traffic and the amount of group B traffic are equal. The A-group traffic partitioning module is used to randomly and equally divide the A-group traffic into at least two subgroup traffic groups; The B-group traffic partitioning module is used to randomly and equally divide the B-group traffic into at least two subgroup traffic, wherein each subgroup of the B-group traffic is equal to each subgroup of the A-group traffic. The subgroup traffic pairing module is used to arrange and combine the subgroup traffic to obtain at least four pairs of subgroup traffic; The AB test module is used to perform multiple comparative tests using the at least four pairs of subgroup traffic to obtain AB test results; The P-value acquisition module is used to perform multiple hypothesis tests during the validity test of the at least four pairs of subgroup traffic to obtain the P-value corresponding to each pair of subgroup traffic. The optimal false discovery rate determination submodule is used to determine the optimal false discovery rate (FDR). The P-value correction submodule is used to correct the P-value corresponding to the flow of each pair of subgroups according to the FDR. The test validity verification module is used to verify the test validity of the at least four pairs of subgroup traffic; The effectiveness verification module is specifically used to verify the effectiveness of the at least four pairs of subgroup traffic. The effectiveness test involves sorting the core indicator values ​​of each subgroup of traffic in group A and each subgroup of traffic in group B according to their mean. If there is an overlap between the sorting results of group A and group B, then at least four pairs of subgroup traffic are subjected to hypothesis testing. If at least two pairs of subgroup traffic have significant differences in their tests, and the directions are opposite, then the core indicator values ​​of group A and group B have significantly overlapped, and the effectiveness test fails; otherwise, the effectiveness test passes.

5. The apparatus according to claim 4, wherein, The test validity verification module is specifically used to perform homogeneity verification on the at least four pairs of subgroup traffic.

6. The apparatus according to claim 4, further comprising: The option setting module is used to set the subgroup traffic option. If the subgroup traffic option is selected, it is applied to the device described in claim 4.

7. An electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-3.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-3.

Images