AB test method and device, computer equipment and medium

Abstract

The embodiment of the invention provides an AB test method and device, computer equipment and a medium, and relates to the technical field of digital data processing.The method comprises the following steps that test data collection modules are implanted into an optimized version of an application program and an original version of the application program respectively, randomly displaying an original version or an optimized version of the application program for a user, and collecting behavior data of the user; determining a P value between the first sample set and the second sample set, and judging whether the AB test is stopped or not according to the statistical significance value; and if the AB test is finished, calculating a target conversion rate of the original version as a first target conversion rate, calculating a target conversion rate of the optimized version as a second target conversion rate, and if the second target conversion rate is higher than the first target conversion rate, taking the optimized version as a formal version of the application program. According to the scheme, the peeping problem is solved by calculating the always effective P value, and the accuracy of the AB test result is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital data processing, and in particular to a method, device, computer equipment and medium for AB testing. Background Art

[0002] AB testing is to develop two plans (such as two pages) for the same goal, let some users use plan A, and another part of users use plan B, record the user's usage, and see which plan is more in line with the design. AB testing has been widely used on the Web, and many testing tools have also appeared.

[0003] One of the most common sources of improper risk management in AB testing is often referred to as "peek". Peek refers to the act of looking at the test results in advance while the AB test is in progress. Looking at the test results in advance can lead to incorrect decisions. This can lead to several serious problems: Increased false positive rate: When we frequently peek at results before a test is complete, we increase the risk of incorrectly believing that there is a difference between two versions, even when there is none. This risk is called a Type I error or a false positive error. Even if the original significance level (such as 5%) is set lower, the actual false positive rate may be significantly higher if we peek at results multiple times during the test. For example, an apparent 1% significance level may actually be a 5% significance level if we peek multiple times during the test. This error can cause us to make decisions based on inaccurate data, which may lead to ineffective or even harmful actions.

[0004] Violation of statistical assumptions: The calculation of the significance level is based on the assumption that the sample size is a predetermined size. If the results are reviewed during the test and the test is terminated early based on these results, this assumption is violated, making the reported significance level meaningless.

[0005] Leading to decision errors: Due to the increased false positive rate caused by "peek", we may mistakenly choose a new version when there is actually no difference between the two versions, which may cause us to abandon the effective old version and adopt a new version that does not perform well. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides an AB testing method to solve the technical problem in the prior art that peeking during AB testing causes errors in the test results. The method includes: Build an optimized version of the application, and implant a test data collection module in the optimized version and the original version of the application respectively. When a user logs into the application, the original version or the optimized version of the application is randomly displayed to the user, and the user's behavior data is collected through the implanted test data collection module. The behavior data of the original version is used as the first sample set, and the behavior data of the optimized version is used as the second sample set. The relationship between the first sample set and the second sample set is determined. P Value, according to P The statistical significance value is calculated and the AB test is stopped according to the statistical significance value. P The value is the probability that the difference between the samples in the first sample set and the second sample set is caused by random factors. P The value continues to decrease, the statistical significance value = 1- P value; If the AB test is over, calculate the target conversion rate of the original version as the first target conversion rate, and calculate the target conversion rate of the optimized version as the second target conversion rate. If the second target conversion rate is higher than the first target conversion rate, use the optimized version as the official version of the application. If the second target conversion rate is lower than the first target conversion rate, rebuild the optimized version and re-conduct AB testing.

[0007] The embodiment of the present invention also provides an AB test device to solve the technical problem in the prior art that peeking during the AB test causes errors in the test results. The device includes: The behavior data collection module is used to build an optimized version of the application. The test data collection module is implanted in the optimized version of the application and the original version of the application respectively. When the user logs into the application, the original version or the optimized version of the application is randomly displayed to the user, and the user's behavior data is collected through the implanted test data collection module; The significance value acquisition test module is used to use the original version of the behavior data as the first sample set and the optimized version of the behavior data as the second sample set to determine the difference between the first sample set and the second sample set. P Value, according to P The statistical significance value is calculated and the AB test is stopped according to the statistical significance value. P The value is the probability that the difference between the samples in the first sample set and the second sample set is caused by random factors. P The value continues to decrease, the statistical significance value = 1- P value; The test result acquisition module is used to calculate the target conversion rate of the original version as the first target conversion rate and the target conversion rate of the optimized version as the second target conversion rate after the AB test is completed. If the second target conversion rate is higher than the first target conversion rate, the optimized version is used as the official version of the application. If the second target conversion rate is lower than the first target conversion rate, the optimized version is rebuilt and the AB test is re-performed.

[0008] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned AB testing methods when executing the computer program, so as to solve the technical problem in the prior art that peeking during the AB test causes errors in the test results.

[0009] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program for executing any of the above-mentioned AB testing methods, so as to solve the technical problem in the prior art that peeking during the AB test causes errors in the test results.

[0010] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: The calculated always valid P Value (i.e., continuously decreasing P value), ensuring that whenever the test is stopped, the P The values ​​are accurate and will not be affected by checking the results in advance (peek), which improves the accuracy of the AB test results. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 is a flow chart of an AB testing method provided by an embodiment of the present invention; Figure 2 is a flow chart of a method for implementing the above AB test provided by an embodiment of the present invention; Figure 3 is a structural block diagram of a computer device provided by an embodiment of the present invention; Figure 4 It is a structural block diagram of an AB testing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0014] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0015] Fixed-horizon testing is a statistical method in which the sample size is determined during the experimental design phase and does not change throughout the experiment. This method is based on frequentist statistics, where p-values ​​and confidence intervals are calculated assuming a fixed sample size.

[0016] A key feature of the fixed visual field test is that it assumes that the experimenter determines the sample size before the experiment begins and does not change it throughout the experiment based on observed data. A limitation of this approach is that it may not be suitable for situations where the sample size needs to be adjusted dynamically based on experimental results.

[0017] In AB testing, fixed horizon testing may not be the best choice because the experimenter may want to adjust the sample size based on the real-time results of the experiment in order to more effectively detect the true effect. Therefore, it is necessary to provide always valid p-values ​​and confidence intervals, allowing the experimenter to dynamically adjust the sample size according to the data during the experiment.

[0018] In an embodiment of the present invention, a method for AB testing is provided, such as Figure 1 and Figure 2 As shown, the method includes: Step S101: construct an optimized version of the application, and implant a test data collection module in the optimized version of the application and the original version of the application respectively. When a user logs into the application, the original version or the optimized version of the application is randomly displayed to the user, and the user's behavior data is collected through the implanted test data collection module; Step S102: The original version of the behavior data is used as the first sample set, and the optimized version of the behavior data is used as the second sample set, and the relationship between the first sample set and the second sample set is determined. P Value, according to P The statistical significance value is calculated and the AB test is stopped according to the statistical significance value. PThe value is the probability that the difference between the samples in the first sample set and the second sample set is caused by random factors. P The value continues to decrease, the statistical significance value = 1- P value; Step S103: If the AB test is completed, the target conversion rate of the original version is calculated as the first target conversion rate, and the target conversion rate of the optimized version is calculated as the second target conversion rate. If the second target conversion rate is higher than the first target conversion rate, the optimized version is used as the official version of the application. If the second target conversion rate is lower than the first target conversion rate, the optimized version is rebuilt and the AB test is repeated.

[0019] To build an optimized version of your application, follow these steps: Analyze the business data of the original version of the application; determine the current improvement points of the test indicators based on the business data of the original version, and establish hypotheses based on the current improvement points, where the hypotheses are feasible solutions to achieve the current improvement points; build an optimized version of the application based on the hypotheses.

[0020] Specifically, the test indicators are the indicators used for reference or consideration in selecting the version of the application in the AB test. According to different actual test requirements, the test indicators can be different indicators. For example, for the test of the application accessed by the user, the current improvement points can be the registration rate, customer retention rate, click-through rate, etc. Applications include computer programs, web programs, and mobile programs. Assume that it is a feasible solution to improve the key improvement points. According to the type of application, embed the test data collection module respectively. If the application is a Web platform, embed the JS script in the page to generate an optimized version of the application. If the application is an APP program, add SDK to generate an optimized version of the application.

[0021] Specifically, when a user logs into the application, a version of any application (original version or optimized version) is randomly selected and displayed to the user, and the visitor's behavior data is collected through the test data collection module. The behavior data includes target conversion data, user access, behavior data, source and exit, and business data.

[0022] In the specific implementation, in order to calculate the statistical significance value and then determine whether to stop the AB test, the following steps are performed to obtain the statistical significance value of the first sample set and the second sample set. P Value, according to P The statistical significance value is calculated by: The target conversion rate is used as the optimization index of the AB test; a definition hypothesis is set for the first sample set and the second sample set, wherein the definition hypothesis includes a null hypothesis and an alternative hypothesis; according to the optimization index and the definition hypothesis, a likelihood ratio is calculated for the first sample set and the second sample set, and the likelihood ratio is used as P value; according to P Calculate the statistically significant value.

[0023] Specifically, define the hypothesis (set the null hypothesis H 0 and the alternative hypothesis H 1 ). Among them, the null hypothesis H 0 is the hypothesis that there is no effect or difference (there is no difference in effect between the original version and the optimized version). The alternative hypothesis H 1 is the hypothesis that an effect or difference exists (the optimized version is more effective than the original version).

[0024] In the single sample case, the test is usually to test the difference between a sample (or a single treatment) and a known benchmark or hypothesis. This test considers independent and identically distributed (iid) observations from an exponential family distribution and uses the likelihood ratio to compare the consistency of the observed data with the null hypothesis. The independent observations are assumed to come from a single sample set. . is a known comparison baseline, is a single sample set, and the value of parameter θ is , P θ represents the probability distribution of the data induced by the parameter θ. In this case, the null hypothesis is , alternative hypothesis .

[0025] Multiple samples are used to compare two or more samples. In this case, it is extended to consider the comparison of multiple samples. Set the null hypothesis , alternative hypothesis .

[0026] In practice, in order to calculate the likelihood ratio to obtain a valid P The following steps are used to calculate the likelihood ratio of the first sample set and the second sample set according to the optimization index and the definition hypothesis: Based on the second sample set, the optimized version of the first average effect estimate is calculated ; Based on the first sample set, the second average effect estimate of the original version is calculated ; Compute the first mean effect estimate and the second mean effect estimate The Difference , where the difference , n is the sample index, is the optimization index of the first sample set, is the optimization index of the second sample set, is the variance, for and The total variance of is the mean difference, is a normal distribution; if the sample set is double, the mean difference between the given mixed distribution H and the null hypothesis is Likelihood ratio ,in, As the sample size increases, the difference The weighting of is the mixed variance; if the sample set is a single sample, the mean difference between the given mixed distribution H and the null hypothesis is Likelihood ratio .

[0027] Specifically, in order to solve the peeking problem, we need to calculate the always valid P Value (i.e., continuously decreasing P If the data are from a normal distribution , and mixed distribution ,but The closed form formula for is: .

[0028] Define the mixture likelihood ratio for the two variant experiments As data and function, for any and (represents the average effect estimate of two different treatment groups. These parameters are used to compare the performance differences between the two groups on a specific indicator (such as click-through rate, conversion rate, satisfaction, etc.). Based on the collected data, calculate the A version and version B ), ,but: ,in, is the mean difference under the null hypothesis.

[0029] For binary data (single sample), Approximately normal distribution , ,but: , where again The difference in success probability under the null hypothesis (if = 0, let = 1).

[0030] Mixed Variance ,in, , a is the significance level, and truncation is the time to end the test. The time to end the test has the same effect as the total number of samples accumulated at the end of the test, so the sample number n value can be used as the truncation.

[0031] In specific implementation, the following steps are performed to reset the saliency: definition P Recursion where the value satisfies: ,in, for P The initial value of the value, is the nth sample P value; according to P The confidence interval I is obtained recursively for the value that satisfies , where , is a variable, is the significance level; when the point estimate When leaving confidence interval I, reset the statistical significance to zero, reset the confidence interval I, and redefine P The value satisfies the recursion, where is the mean of all samples in the first sample set, is the mean of all samples in the second sample set.

[0032] Specifically, the monotonicity of the confidence interval I can present some challenges. In some cases, the confidence interval I may lie completely above or below the true difference, and even if the point estimate (point estimate) gradually approaches the true value, and the confidence interval I will not return to normal. This phenomenon may be exacerbated by changes in the base conversion rate during the experiment. In order to solve the monotonicity of the confidence interval, a heuristic strategy for resetting significance is proposed. Leaving the confidence interval I forces a reset of the inference: the reported statistical significance is reset to zero, the confidence interval reverts to the entire real number line, and then iteratively redefines P The recursion restarts when the value is satisfied. Using the above method will make the p value larger and the confidence interval I wider, without increasing the false positive error, but may increase the false negative error.

[0033] In order to solve the error caused by increased false negatives, the maximum waiting sample size M is set to reduce the waiting time and consume less sample size. In this case, truncation to M achieves significance earlier on average than competing tests providing the same power.

[0034] In specific implementation, in order to stop AB testing when business requirements are met, the following steps are used to determine whether to stop AB testing based on statistical significance values: Set the significance threshold; if the statistical significance value is greater than or equal to the significance threshold, end the current AB test; if the statistical significance value is less than the significance threshold, continue the current AB test.

[0035] Specifically, the embodiment of the present invention calculates P The value is always valid P The statistical significance value will continue to increase with the test time and will not decrease. Therefore, you only need to set the significance threshold according to actual business requirements. When it is greater than the significance threshold, you can stop the AB test.

[0036] In specific implementation, in order to display the target conversion rate on the display interface, the target conversion rate of the original version is calculated as the first target conversion rate through the following steps: The number of times the user uses the original version of the interface to be tested is obtained from the behavior data as the number of visits; the number of times the user triggers any target to be tested in the original version of the interface to be tested is obtained from the behavior data as the number of conversions; the first target conversion rate is calculated by the number of conversions and the number of visits, wherein the first target conversion rate = the number of conversions / the number of visits.

[0037] Specifically, the calculation method of the second target conversion rate is the same as that of the second target conversion rate. The target conversion rates of different versions are compared, and the ratio of the target conversion rate of the optimized version to the original version is obtained through the conversion rate ratio, and output to the display interface.

[0038] Specifically, after the AB test is completed, the data indicators such as the number of independent visitors, number of visits, number of conversions, target conversion rate, etc. of the original version and the optimized version can be counted to generate statistical results.

[0039] Calculate the number of unique visitors. The number of unique visitors is the number of users or devices that access a web application or app within a certain period of time. For the number of unique visitors of a web application, users who are not logged in can determine whether the user is a unique visitor based on the user's browser cookies, IP address, device resolution, browser model, operating system and other information. For example, if a user uses different browsers or different devices to access an application within a certain period of time, it will be recorded as multiple unique visitors. For logged in users, the number of unique visitors can be identified by user ID.

[0040] The number of unique visitors to the APP application. If the user is not logged in, the model, operating system, brand, resolution, IP address and other information of the user's mobile device can be used to determine whether the user is a unique visitor. For logged in users, the number of unique visitors can be identified by user ID.

[0041] Finally, do business analysis on the AB test results. Business analysis is to find the real reason why the optimized version is better or worse than the original version, which is an important part of AB test result analysis. Based on the business domain knowledge of the industry you are engaged in, combined with customer psychology, usage habits, intuition, instinct, preference and other perspectives, dig out the truth behind the numbers to better guide future actions.

[0042] In this embodiment, a computer device is provided, such as Figure 3 As shown, it includes a memory 301, a processor 302, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, any of the above-mentioned AB testing methods is implemented.

[0043] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0044] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned AB testing methods.

[0045] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0046] Based on the same inventive concept, an AB testing device is also provided in the embodiments of the present invention, as described in the following embodiments. Since the principle of solving the problem by the AB testing device is similar to that of the AB testing method, the implementation of the AB testing device can refer to the implementation of the AB testing method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0047] Figure 4 is a structural block diagram of an AB test device according to an embodiment of the present invention, such as Figure 4 As shown, it includes: a behavior data collection module 401, a significance value acquisition test module 402 and a test result acquisition module 403. The structure is described below.

[0048] The behavior data collection module 401 is used to build an optimized version of the application, and to implant a test data collection module in the optimized version of the application and the original version of the application respectively. When a user logs into the application, the original version or the optimized version of the application is randomly displayed to the user, and the user's behavior data is collected through the implanted test data collection module; The significance value acquisition test module 402 is used to use the original version of the behavior data as the first sample set and the optimized version of the behavior data as the second sample set to determine the significance value between the first sample set and the second sample set. P Value, according to P The statistical significance value is calculated and the AB test is stopped according to the statistical significance value. P The value is the probability that the difference between the samples in the first sample set and the second sample set is caused by random factors. P The value continues to decrease, the statistical significance value = 1- P value; The test result acquisition module 403 is used to calculate the target conversion rate of the original version as the first target conversion rate and the target conversion rate of the optimized version as the second target conversion rate after the AB test is completed. If the second target conversion rate is higher than the first target conversion rate, the optimized version is used as the official version of the application. If the second target conversion rate is lower than the first target conversion rate, the optimized version is rebuilt and the AB test is performed again.

[0049] In one embodiment, the behavior data collection module includes: A business analysis data unit, used to analyze the business data of the original version of the application; Establish a hypothesis unit to determine the current improvement point of the test indicator based on the original version of the business data, and establish a hypothesis based on the current improvement point, where the hypothesis is a feasible solution to achieve the current improvement point; Optimized version build unit, used to build an optimized version of the application based on assumptions.

[0050] In one embodiment, the significance value acquisition test module includes: Determine the optimization indicator unit, which is used to use the target conversion rate as the optimization indicator for AB testing; A definition hypothesis unit, used for setting definition hypotheses for the first sample set and the second sample set, wherein the definition hypothesis includes a null hypothesis and an alternative hypothesis; A likelihood ratio calculation unit is used to calculate the likelihood ratio of the first sample set and the second sample set according to the optimization index and the definition hypothesis, and use the likelihood ratio as P value; Calculate the significance value unit for P Calculate the statistically significant value.

[0051] In one embodiment, the likelihood ratio calculation unit is used to calculate the optimized version of the first average effect estimate based on the second sample set. ; Based on the first sample set, the second average effect estimate of the original version is calculated ; Compute the first mean effect estimate and the second mean effect estimate The Difference , where the difference , n is the sample index, is the optimization index of the first sample set, is the optimization index of the second sample set, is the variance, for and The total variance of is the mean difference, is a normal distribution; if the sample set is a two-sample, the mean difference between the given mixed distribution H and the null hypothesis is Likelihood ratio ,in, As the sample size increases, the difference The weighting of is the mixed variance; if the sample set is a single sample, the mean difference between the given mixed distribution H and the null hypothesis is Likelihood ratio .

[0052] In one embodiment, the significance value acquisition test module further includes: Reset saliency unit, used to define P Recursion where the value satisfies: ,in, for P The initial value of the value, is the nth sample P value; according to the recursion of P value to obtain the confidence interval I, where, , is a variable, is the significance level; when the point estimate When leaving confidence interval I, reset the statistical significance to zero, reset the confidence interval I, and redefine P The value satisfies the recursion, where is the mean of all samples in the first sample set, is the mean of all samples in the second sample set.

[0053] In one embodiment, the significance value acquisition test module further includes: A threshold setting unit, used to set a significance threshold; The end test judgment unit is used to end the current AB test if the statistical significance value is greater than or equal to the significance threshold; The test continuing judgment unit is used to continue the current AB test if the statistical significance value is less than the significance threshold.

[0054] In one embodiment, the significance value acquisition test module further includes: A visit number acquisition unit, used to acquire the number of times the user uses the original version of the test interface from the behavior data as the visit number; A conversion number acquisition unit, used to acquire the number of times that a user triggers any target to be tested in the original version of the test interface from the behavior data as the conversion number; The target conversion rate calculation unit is used to calculate a first target conversion rate through the conversion number and the visit number, wherein the first target conversion rate=conversion number / visit number.

[0055] The embodiments of the present invention achieve the following technical effects: The calculated always valid P value, ensuring that whenever the test is stopped, the P The values ​​are accurate and will not be affected by looking at the results in advance (peek); by resetting the significance, PThe value is larger and the confidence interval is wider, which will not increase the false positive errors. For the possible increase of false negative errors, the false negative errors can be reduced by setting the maximum waiting sample size M. In order to find the real reason why the optimized version is better or worse than the original version, the AB test results can be analyzed from business perspectives based on the business field knowledge of the industry in which you are engaged, combined with customer psychology, usage habits, intuition, instinct, preference tendencies, etc., which can effectively explore the reasons behind the numbers and use them to better guide future actions.

[0056] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method of AB testing, characterized in that: include: Constructing an optimized version of an application, implanting a test data collection module in the optimized version of the application and the original version of the application respectively, and when a user logs into the application, randomly displaying the original version or the optimized version of the application to the user, and collecting user behavior data through the implanted test data collection module; The behavior data of the original version is used as a first sample set, and the behavior data of the optimized version is used as a second sample set, and the relationship between the first sample set and the second sample set is determined. P value, according to the P The value is calculated to obtain a statistically significant value, and whether to stop the AB test is determined according to the statistically significant value, wherein the P The value is the probability that the difference between the samples in the first sample set and the second sample set is caused by random factors. P The value continues to decrease, the statistical significance value = 1-the P value; If the AB test is finished, the target conversion rate of the original version is calculated as the first target conversion rate, and the target conversion rate of the optimized version is calculated as the second target conversion rate. If the second target conversion rate is higher than the first target conversion rate, the optimized version is used as the official version of the application. If the second target conversion rate is lower than the first target conversion rate, the optimized version is rebuilt and the AB test is performed again.

2. The AB testing method according to claim 1, characterized in that: The first sample set and the second sample set are statistically analyzed to obtain a constant effective P value, according to the P Statistically significant values ​​were calculated using the following formula: Using the target conversion rate as an optimization indicator for AB testing; Setting definition hypotheses for the first sample set and the second sample set, wherein the definition hypotheses include a null hypothesis and an alternative hypothesis; According to the optimization index and the definition hypothesis, a likelihood ratio is calculated for the first sample set and the second sample set, and the likelihood ratio is used as the P value; According to the P Calculate the statistically significant value.

3. The AB testing method according to claim 2, characterized in that: Calculating a likelihood ratio for the first sample set and the second sample set according to the optimization index and the definition hypothesis includes: Based on the second sample set, a first average effect estimate of the optimized version is calculated ; Based on the first sample set, a second average effect estimate of the original version is calculated ; Calculate the first mean effect estimate and the second average effect estimate The Difference , where the difference , n is the sample index, is the optimization index of the first sample set, is the optimization index of the second sample set, is the variance, for and The total variance of is the mean difference, is a normal distribution; If the sample set is a two-sample, the mean difference between the given mixed distribution H and the null hypothesis is Likelihood ratio ,in, As the sample size increases, the difference The weighting of is the mixed variance; If the sample set is a single sample, the mean difference between the given mixed distribution H and the null hypothesis is Likelihood ratio .

4. The AB testing method according to claim 3, characterized in that: Also includes: definition P Recursion where the value satisfies: ,in, For the P The initial value of the value, is the nth sample P value; According to the P The confidence interval I is obtained recursively for the value that satisfies , where , is a variable, is the significance level; When the point estimate When leaving the confidence interval I, reset the statistical significance to zero, reset the confidence interval I, and redefine the P The value satisfies the recursion, where is the mean of all samples in the first sample set, is the mean of all samples in the second sample set.

5. The AB testing method according to claim 1, characterized in that: Determining whether to stop the AB test according to the statistical significance value includes: Set significance threshold; If the statistical significance value is greater than or equal to the significance threshold, the current AB test is terminated; If the statistical significance value is less than the significance threshold, the current AB test is continued.

6. The AB testing method according to any one of claims 1 to 5, characterized in that: Build an optimized version of your application, including: Analyze business data of the original version of the application; Determine the current improvement point of the test indicator according to the business data of the original version, and establish a hypothesis according to the current improvement point, wherein the hypothesis is a feasible solution to achieve the current improvement point; An optimized version of the application is constructed based on the assumptions.

7. The AB testing method according to any one of claims 1 to 5, characterized in that: Calculating the target conversion rate of the original version as the first target conversion rate includes: Acquire the number of times the user uses the original version of the interface to be tested from the behavior data as the number of visits; Acquire, from the behavior data, the number of times the user triggers any target to be tested in the original version of the interface to be tested as the number of conversions; A first target conversion rate is calculated by using the conversion number and the visit number, wherein the first target conversion rate=the conversion number / the visit number.

8. An AB testing device, characterized in that: include: A behavior data collection module, used to construct an optimized version of the application, and to implant a test data collection module in the optimized version of the application and the original version of the application respectively, so that when a user logs into the application, the original version or the optimized version of the application is randomly displayed to the user, and the user's behavior data is collected through the implanted test data collection module; The significance value acquisition test module is used to use the behavior data of the original version as a first sample set and the behavior data of the optimized version as a second sample set to determine the significance between the first sample set and the second sample set. P value, according to the P The value is calculated to obtain a statistically significant value, and whether to stop the AB test is determined according to the statistically significant value, wherein the P The value is the probability that the difference between the samples in the first sample set and the second sample set is caused by random factors. P The value continues to decrease, the statistical significance value = 1-the P value; The test result acquisition module is used to calculate the target conversion rate of the original version as the first target conversion rate and the target conversion rate of the optimized version as the second target conversion rate when the AB test is completed. If the second target conversion rate is higher than the first target conversion rate, the optimized version is used as the official version of the application. If the second target conversion rate is lower than the first target conversion rate, the optimized version is rebuilt and the AB test is performed again.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the AB testing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the AB testing method according to any one of claims 1 to 7.

Images