Business significant threshold determination method, business experiment method and business significant threshold determination device

Abstract

The invention provides a method for determining a service significant threshold, and relates to the technical field of computers, in particular to the technical field of online experiments. According to the specific implementation scheme, a confidence interval expression of index differences is obtained; the index difference represents the difference of the business index in the experiment of the first object; according to control group data obtained by performing a business index experiment on the first object, determining a statistical magnitude; based on the statistics and the confidence interval expression, determining a service significance threshold corresponding to the service index, the service significance threshold being used for determining the service significance of an experimental result corresponding to the service index in the service experiment, thereby improving the reliability of the service experiment; the invention further provides a service experiment method and device, electronic equipment, a storage medium and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of online experiment technology in computer technology, and in particular to a method for determining a business significance threshold, a business experiment method and a device. Background Art

[0002] Business experiments are online experiments on different versions of products (such as applications and web pages). The experimental results are determined by the performance of different versions of products on business indicators, and the significance of the experimental results is statistically analyzed to evaluate the business significance of the experimental results.

[0003] At present, there is a problem that the business is statistically significant, but in fact the performance of business indicators is slightly different. How to improve the reliability of business experiments is an urgent problem to be solved. Summary of the invention

[0004] The present disclosure provides a method for determining a business significance threshold, a business experiment method and a device.

[0005] According to a first aspect of the present disclosure, a method for determining a business significance threshold is provided, comprising: obtaining a confidence interval expression of an indicator difference; the indicator difference represents the difference of the business indicator in an experiment of a first object; determining a statistic based on control group data obtained from an experiment of the business indicator on the first object; and determining a business significance threshold corresponding to the business indicator based on the statistic and the confidence interval expression, the business significance threshold being used to determine the business significance of the experimental results corresponding to the business indicator in the business experiment.

[0006] According to a second aspect of the present disclosure, a business experiment method is provided, including: obtaining control group data obtained by conducting an experiment on a business indicator on a first object; conducting an experiment on the business indicator on a second object to obtain experimental group data; the second object is a different version from the first object; based on a business significance threshold, the control group data and the experimental group data are compared to obtain experimental results of the business indicator; the business significance threshold is determined based on the method of the first aspect.

[0007] According to a third aspect of the present disclosure, a device for determining a service significance threshold is provided, including:

[0008] An acquisition unit, used for acquiring a confidence interval expression of an indicator difference; the indicator difference represents a difference of the business indicator in the experiment of the first object;

[0009] A statistic determination unit, configured to determine a statistic based on control group data obtained from an experiment on a business indicator of a first object;

[0010] The business significance threshold determination unit is used to determine the business significance threshold corresponding to the business indicator based on the statistic and the confidence interval expression.

[0011] According to a fourth aspect of the present disclosure, a service experiment device is provided, including:

[0012] A first experimental unit is used to conduct a business indicator experiment on the first object to obtain control group data of the business indicator;

[0013] The second experimental unit is used to perform an experiment on a business indicator on a second object to obtain experimental group data of the business indicator; the second object is a different version from the first object;

[0014] The business significance determination unit is used to determine the experimental results corresponding to the business indicators based on the control group data and the experimental group data, and determine the business significance of the experimental results according to the business significance threshold; the business significance threshold is determined according to the method of the first aspect.

[0015] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods of the first aspect and the second aspect.

[0016] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the methods of the first aspect and the second aspect.

[0017] According to the seventh aspect of the present disclosure, a computer program product is provided, including: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the methods of the first aspect and the second aspect.

[0018] The technology disclosed in the present invention improves the reliability of business experiments.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 The following is a flow chart of a method for determining a service significance threshold provided by an embodiment of the present disclosure. Figure 1 ;

[0022] Figure 2The following is a flow chart of a method for determining a service significance threshold provided by an embodiment of the present disclosure. Figure 2 ;

[0023] Figure 3 The probability density curve corresponding to the scale index provided in the embodiment of the present disclosure is shown as follows: Figure 1 ;

[0024] Figure 4 The probability density curve corresponding to the scale index provided in the embodiment of the present disclosure is shown as follows: Figure 2 ;

[0025] Figure 5 The probability density curve corresponding to the scale index provided in the embodiment of the present disclosure is shown as follows: Figure 3 ;

[0026] Figure 6 The probability density curve corresponding to the scale index provided in the embodiment of the present disclosure is shown as follows: Figure 4 ;

[0027] Figure 7 The probability density curve corresponding to the scale index provided in the embodiment of the present disclosure is shown as follows: Figure 5 ;

[0028] Figure 8 The probability density curve corresponding to the scale index provided in the embodiment of the present disclosure is shown as follows: Figure 6 ;

[0029] Fig. 9 The probability density curve corresponding to the scale index provided in the embodiment of the present disclosure is shown as follows: Figure 7 ;

[0030] Fig.10 The probability density curve corresponding to the ratio index provided in the embodiment of the present disclosure is shown as follows: Figure 1 ;

[0031] Fig.11 The probability density curve corresponding to the ratio index provided in the embodiment of the present disclosure is shown as follows: Figure 2 ;

[0032] Fig.12 The probability density curve corresponding to the ratio index provided in the embodiment of the present disclosure is shown as follows: Figure 3 ;

[0033] Fig.13 The probability density curve corresponding to the ratio index provided in the embodiment of the present disclosure is shown as follows: Figure 4 ;

[0034] Fig.14 The probability density curve corresponding to the ratio index provided in the embodiment of the present disclosure is shown as follows: Figure 5 ;

[0035] Fig.15 The probability density curve corresponding to the ratio index provided in the embodiment of the present disclosure is shown as follows: Figure 6 ;

[0036] Fig.16 The probability density curve corresponding to the ratio index provided in the embodiment of the present disclosure is shown as follows: Figure 7 ;

[0037] Fig.17 It is a flowchart of a business experiment method provided by an embodiment of the present disclosure;

[0038] Fig.18 It is a structural schematic diagram of a device for determining a service significance threshold provided by an embodiment of the present disclosure;

[0039] Fig.19 It is a structural diagram of a business experiment device provided by an embodiment of the present disclosure;

[0040] Fig. 20 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0042] Business experiments are online experiments on different versions of products (such as applications and web pages). The experimental results are determined by the performance of different versions of products in business indicators, and the significance of the experimental results is statistically analyzed to evaluate the reliability of the experimental results.

[0043] Usually, an expected value is set as the business significance threshold based on the needs of the business experiment or based on experience to evaluate whether the experimental results meet the business expectations or the degree of improvement. However, the performance of different business indicators in different business scenarios and on different versions of products varies greatly. The set business significance threshold is difficult to apply to the business significance evaluation of different business scenarios and different business indicators. It is easy to have the problem that the experimental results are significantly significant and the business is determined to be significant, but the actual business performance of the business indicator is slightly different, resulting in low reliability of the business experiment.

[0044] The present disclosure provides a method for determining a business significance threshold, a business experiment method and a device, which are applied to the field of online experiments in computer technology to achieve the effect of improving the reliability of business experiments.

[0045] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0046] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 1-Figure 20 The embodiments of the present disclosure are further refined.

[0047] Figure 1 A flow chart of a method for determining a business significance threshold provided by an embodiment of the present disclosure is shown as follows: Figure 1 , the method comprising:

[0048] S101. Obtain a confidence interval expression of an indicator difference; the indicator difference represents a difference in a business indicator in an experiment of the first object.

[0049] The execution subject of the embodiment of the present disclosure may be a device for determining a service significance threshold, and the device for determining a service significance threshold may be coupled to a server.

[0050] Among them, the first object is the first version of the product, such as the first version of the application, web page, etc.; business indicators are used to reflect the user participation in the product, such as business indicators can be click volume, activity, satisfied consumption rate, retention rate, etc.

[0051] The experiment of business indicators on the first object refers to executing an experiment in which different users use the first object under the same experimental configuration, and statistically analyzing the differences in business indicators in the experiment; the indicator difference means the difference in business indicators in the experiment; because in the experiment, the experimental configuration is the same and users all use the first object, the indicator difference can reflect the natural fluctuation of the business indicators in the experiment.

[0052] Exemplarily, users are randomly divided into a first control group and a second control group. Users in the first control group use the first object, and the business indicators of the first control group are counted. Users in the second control group also use the first object, and the business indicators of the second control group are counted. The difference between the business indicators of the first control group and the business indicators of the second control group is the indicator difference.

[0053] The confidence interval expression of the indicator difference is the expression for calculating the confidence interval of the indicator difference. The confidence interval can be used to calculate the business significant difference. Since the indicator difference can reflect the natural fluctuation of the business indicator, the confidence interval of the indicator difference can represent the natural fluctuation range of the indicator difference in business performance. If the indicator difference in the actual experiment exceeds the confidence interval, it means that the indicator difference in business performance exceeds the natural fluctuation range. Therefore, it can be considered that the indicator difference is significant in business performance.

[0054] Optionally, different types of business indicators correspond to different confidence interval expressions; for different types of business indicators, confidence interval expressions corresponding to the indicator differences of business indicators of that type are pre-constructed; for the business indicators of the current experiment, the type of business indicator is obtained, and the confidence interval expression corresponding to the indicator difference is obtained based on the type of business indicator.

[0055] Optionally, the types of business indicators include but are not limited to scale indicators and ratio indicators; scale indicators involve one indicator, such as clicks, views, etc.; ratio indicators involve two indicators, which can be the ratio between the two indicators, such as the satisfied consumption rate is the proportion of satisfied consumption in the total consumption, and the retention rate is the ratio between the number of active users on the next day and the number of new users on the same day.

[0056] S102: Determine a statistic based on control group data obtained from a business indicator experiment on the first object.

[0057] The statistic is calculated based on the sample data of the indicator difference of the business indicator, and is used to describe the characteristics of the indicator difference of the business indicator. In this embodiment, the statistic may include but is not limited to: sample size, sample mean and sample variance.

[0058] Optionally, performing an experiment on a business indicator on the first object includes performing a first experiment and a second experiment on the business indicator on the first object; and determining a statistic according to experimental data obtained from the first experiment and experimental data obtained from the second experiment.

[0059] Optionally, conducting a first experiment and a second experiment on a business indicator on a first object refers to randomly dividing users into a first control group and a second control group, so that users in the first control group use the first object, and users in the second control group use the first object; obtaining the indicator value of the business indicator when users in the first control group use the first object, that is, the experimental data of the first experiment, and obtaining the indicator value of the business indicator when users in the second control group use the first object, that is, the experimental data of the second experiment.

[0060] Optionally, the statistic is determined based on the experimental data obtained from the first experiment and the experimental data obtained from the second experiment. The first sample size, the first sample mean and the first sample variance are determined based on the experimental data obtained from the first experiment, and the second sample size, the second sample mean and the second sample variance are determined based on the experimental data obtained from the second experiment. Since the first experiment and the second experiment are parts of the experiment on the first object, the experimental environments are the same. It can be considered that the first sample size and the second sample size are the same, the first sample mean and the second sample mean are the same, and the first sample variance and the second sample variance are the same. Furthermore, the first sample size (or the second sample size) can be used as the sample size, the first sample mean (or the second sample mean) can be used as the sample mean, and the first sample variance (or the second sample variance) can be used as the sample variance. Alternatively, the sample size, sample mean and sample variance can be determined by combining the experimental data of the first experiment and the experimental data of the second experiment.

[0061] S103. Determine a business significance threshold corresponding to the business indicator based on the statistic and the confidence interval expression; the business significance threshold is used to determine the business significance of the experimental result corresponding to the business indicator in the business experiment.

[0062] Optionally, the statistics include sample size, sample mean and sample variance, and the confidence interval expression includes sample size parameter, expected parameter and variance parameter; based on the consistency of the estimator, it can be considered that when the sample size tends to infinity, the sample mean converges to the expected mean of the population with probability, and the sample variance converges to the variance of the population with probability; therefore, the sample size, sample mean and sample variance are substituted into the confidence interval expression to obtain the confidence interval, and the business significance threshold of the business indicator is determined based on the confidence interval.

[0063] In the business experiment, a business indicator experiment is conducted on the first object to obtain control group data, and a business indicator experiment is conducted on the second object to obtain experimental group data. The difference in business indicators is determined based on the control group data and the experimental group data, and the experimental results are determined based on the difference in business indicators. The business significance of the experimental results is determined through a business significance threshold.

[0064] Optionally, the experimental results can represent the degree of improvement of the business performance of the business indicator on the second object compared to the business performance on the first object; since the indicator difference is the difference of the business indicator in the experiment of the first object, the confidence interval expression of the indicator difference can reflect the natural fluctuation range of the indicator difference. Therefore, based on the business significance threshold, it can be determined whether the improvement degree belongs to the natural fluctuation range or is an actual improvement in business benefits, that is, the business significance of the experimental results can be determined; for example, if the improvement degree does not exceed the business significance threshold, the improvement may be caused by the natural fluctuation of the business indicator in the experiment, and the experimental result is not significant in business; if the improvement degree exceeds the business significance threshold, the improvement may be caused by the actual business benefits, that is, the experimental result is significant in business.

[0065] The method for determining a business significance threshold provided by the embodiment of the present disclosure obtains a confidence interval expression of an indicator difference, determines a statistic based on control data obtained by conducting an experiment on a business indicator of a first object, substitutes the statistic into the confidence interval expression, and obtains a business significance threshold corresponding to the business indicator; since the indicator difference represents the difference of the business indicator in the experiment of the first object, that is, the confidence interval expression of the indicator difference, can reflect the natural fluctuation range of the indicator difference, and then the business significance threshold determined by the statistic determined by the experiment and the confidence interval expression of the indicator difference can determine whether the experimental result is caused by natural fluctuation or actual difference in the business, that is, the business significance threshold can be used to determine the significance of the business indicator in the business, avoiding the situation where the experimental result is significant but there is no actual improvement in the business, thereby improving the reliability of the business experiment; in addition, by determining the statistic based on the control data obtained by the experiment on the first object, and calculating the business significance threshold based on the statistic and the confidence interval expression, less data is involved, and the amount of calculation is small, so the business significance threshold can be obtained quickly, thereby improving the efficiency of determining the business significance threshold.

[0066] Optionally, based on any of the above embodiments, before obtaining the confidence interval expression of the indicator difference, the method also includes: determining the asymptotic distribution of the indicator difference; and constructing a confidence interval expression of the indicator difference under a preset confidence level based on the asymptotic distribution of the indicator difference.

[0067] In this embodiment, when the sample size of the indicator difference tends to infinity, the distribution of the sample statistic tends to a stable distribution, that is, the asymptotic distribution; optionally, the asymptotic distribution of the random variable difference is obtained, and the random variable difference is the difference between two random variables, corresponding to the difference between the two parts of the control group data in the first object experiment of the business indicator. Therefore, based on the asymptotic distribution of the random variable difference, the asymptotic distribution of the indicator difference is derived.

[0068] Optionally, the asymptotic distribution is a normal distribution, and the quantile corresponding to a preset confidence level is obtained. Based on the quantile and the variance expression of the normal distribution, a confidence interval expression of the indicator difference under the preset confidence level is constructed.

[0069] For example, the preset confidence level is 95%, and the quantile of 95% confidence level is determined to be 1.96 through the normal distribution table. The variance expression of the normal distribution is obtained as ∑. According to the quantile and the variance expression of the normal distribution, the confidence interval expression is determined, such as

[0070] In this embodiment, a confidence interval expression of the indicator difference is constructed through the asymptotic distribution of the indicator difference, so that the business significance threshold can be calculated through the confidence interval expression later, thereby improving the credibility of the business experiment.

[0071] Optionally, based on any of the above embodiments, the asymptotic distribution of indicator differences is determined, including: obtaining the asymptotic distribution of random variable differences; the random variable difference represents the difference between a first random variable and a second random variable, and the variance of the asymptotic distribution is determined based on the mean and variance of the first random variable and the second random variable; using first control data obtained from a first experiment on a business indicator of a first object as the first random variable; using second control data obtained from a second experiment on a business indicator of the first object as the second random variable; based on the asymptotic distribution of random variable differences, when the expectation and variance of the first control data are respectively equal to the expectation and variance of the second control data, obtaining the asymptotic distribution of indicator differences; based on the asymptotic distribution of random variable differences, when the mean and variance of the first control data are respectively equal to the mean and variance of the second control data, obtaining the asymptotic distribution of indicator differences.

[0072] In this embodiment, the asymptotic distribution of the difference of random variables is determined by the Delta Method theorem; Delta Method is an approximate method for deriving the functional distribution of random variables. The Delta Method theorem states: for X1, X2, …, X n , where X i is a k-dimensional random variable, if a n (X n -c) can converge to a k-dimensional normal distribution N according to the distribution k (0,∑), then Among them, lim n→∞ a n =∞, and the function g is a measurable function, g: is the gradient operator. For a function g(x1,x2,…,x k ), This is a k-dimensional vector.

[0073] For independent and identically distributed random variables (X1, Y1),…, (X n ,Y n ), X i is the first random variable, Y i is the second random variable, X i The expectation is μ x , the variance is Y i The expectation is μ y , the variance is The covariance of the first random variable and the second random variable is σ xy ; The random variable difference is The expectation of the difference of the random variables is By using the DeltaMethod theorem, the asymptotic distribution of the difference of random variables can be determined, as shown in formula (1).

[0074] Formula (1):

[0075] It can be seen that n represents the sample size of the random variable difference, and the variance of the asymptotic distribution of the random variable difference is The variance of the asymptotic distribution is determined based on the expectation and variance of each of the first random variable and the second random variable.

[0076] The first control data P1 obtained by performing a first experiment on the business indicator of the first object is used as the first random variable, the second control data P2 obtained by performing a second experiment on the business indicator of the first object is used as the second random variable, and the indicator difference between the first control data and the second control data is used as the random variable difference, and the asymptotic distribution of the indicator difference can be obtained: as shown in formula (2).

[0077] Formula (2):

[0078] Among them, r n is the indicator difference, r n Can be μ r is the expectation of the indicator difference, μ1 is the expectation of the first control data, is the variance of the first control data, μ2 is the expectation of the second control data, is the variance of the second control data, σ 12 is the covariance of the first control data and the second control data.

[0079] It should be noted that the experiment of business indicators on the first object can be a control group experiment in the business experiment; specifically, the business experiment can be an AB experiment, and the AB experiment refers to randomly dividing users into a control group (Group A) and an experimental group (Group B), the control group uses the first object, and the control group experimental data is counted, the experimental group uses the second object, and the experimental group data is counted, and the experimental results are determined according to the control group data and the experimental group data; this embodiment constructs a confidence interval expression based on the characteristics of the control group experiment. Specifically, the users of the control group are randomly divided into Group A1 and Group A2, and the first control data is obtained by the users of Group A1 using the first object, and the second control data is obtained by the users of Group A2 using the first object. Therefore, the experiment of business indicators on the first object can be called an AA experiment.

[0080] Since in the AA experiment, the first control data and the second control data are both obtained by conducting experiments on the business indicators of the first object, the expectation and variance of the first control data and the second control data can be set equal, and then the asymptotic distribution of the indicator difference can be determined according to formula (3).

[0081] Formula (3):

[0082] Among them, r n represents the indicator difference, n represents the sample size, μ represents the expectation of the control data obtained by the experiment of the business indicator of the first object, σ 2 Represents the variance of the control data obtained from the experiment of the business indicator on the first object.

[0083] At a confidence level of 95%, the confidence interval expression of the indicator difference is The asymptotic distribution of the indicator difference is It can be obtained that at a confidence level of 95%, the confidence interval expression of the indicator difference is:

[0084] Next, we will explain the process of determining the asymptotic distribution of the difference of random variables through the Delta Method theorem. Determined through the Delta Method theorem: For independent and identically distributed random variables (X1, Y1),…, (X n ,Y n ), X i is the first random variable, Y i is the second random variable, X i The expectation is μ x , the variance is Y i The expectation is μ y , the variance is The covariance of the first random variable and the second random variable is σxy ; The random variable difference is According to the central limit theorem, we can get:

[0085] Right now

[0086] Hope to get The asymptotic distribution of Then there exists:

[0087] The calculation can be obtained:

[0088]

[0089] Applying the Delta Method gives:

[0090]

[0091] In this embodiment, the confidence interval expression of the indicator difference in the AA experiment is derived through the asymptotic distribution of the random variable difference, so that the business significance threshold can be calculated through statistics and confidence interval expressions in the future, thereby improving the efficiency of calculating the business significance threshold and improving the credibility of the business experiment.

[0092] Optionally, based on any of the above embodiments, a statistic is determined based on the control group data obtained from an experiment on a business indicator of the first object, including: obtaining the control group data obtained from an experiment on a business indicator of the first object; the control group data includes multi-bucket control data; using the number of buckets of the multi-bucket control data as the sample size of the business indicator; determining the sample mean and sample variance of the business indicator based on the multi-bucket control data; the statistic includes the sample size, sample mean and sample variance.

[0093] The control group data may be indicator values ​​of business indicators.

[0094] In this embodiment, the control group data obtained by performing a business indicator experiment on the first object includes first control data P1 and second control data P2. In the AA experiment, bucketing is performed in the user dimension, and the first control data can be expressed as The second control data can be expressed as n is the number of buckets, and the indicator difference is shown in formula (4).

[0095] Formula (4):

[0096]

[0097] Since the first control data and the second control data are both obtained by conducting business indicator experiments on the first object, the expectation and variance of the first control data and the second control data can be set equal, that is, each bucket of data in the first control data can be merged with each bucket of data in the second control data to obtain multi-bucket control data of the control group data, for example, P is the control group data (the sum of multiple barrel control data), p i It is the comparison data of the i-th bucket. The number of buckets is used as the sample size. The sample mean and sample variance of the business indicator are determined based on the multi-bucket comparison data.

[0098] In this embodiment, the sample size, sample mean and sample variance of the business indicator are determined by using the control group data obtained by conducting an experiment on the business indicator of the first object, without the need for multiple resampling, thereby reducing the number of experiments and the amount of calculation required to calculate the business significance threshold, and improving the efficiency of calculating the business significance threshold.

[0099] Optionally, based on any of the above embodiments, the business indicator includes a scale indicator; determining the sample mean and sample variance of the business indicator based on the multi-bucket control data includes: determining the average value of the multi-bucket control data, and using the average value as the sample mean of the scale indicator; determining the variance of the multi-bucket control data based on the multi-bucket control data and the average value, and using the variance as the sample variance of the scale indicator.

[0100] Among them, the scale indicator involves an indicator, such as clicks, views, etc.

[0101] In this embodiment, the sum of the multi-bucket comparison data of the scale indicator is: Determine the mean of multiple buckets of control data: And determine the variance s based on the mean and multi-bucket control data 2 , taking the mean of the scale indicators as the sample mean and the variance as the sample variance.

[0102] In this embodiment, the sample mean and sample variance of the scale indicator are determined based on multi-bucket comparison data of the scale indicator, without the need for multiple resampling, thereby reducing the number of experiments and the amount of calculation required to calculate the business significance threshold, and improving the efficiency of calculating the business significance threshold.

[0103] Optionally, based on any of the above embodiments, the business indicator includes a ratio indicator, which is the ratio between a first indicator and a second indicator; determining the sample mean and sample variance of the business indicator based on multi-bucket comparison data, including: determining a first mean and a first variance of the first indicator, a second mean and a second variance of the second indicator, and a covariance between the first indicator and the second indicator based on the multi-bucket comparison data; determining the sample mean of the ratio indicator based on the first mean and the second mean; determining the sample variance of the ratio indicator based on the first mean, the sample mean, the first variance, the second variance and the covariance.

[0104] Among them, the proportion indicator involves the first indicator and the second indicator. The proportion indicator is the ratio between the first indicator and the second indicator. For example, the satisfied consumption rate is the proportion of satisfied consumption in the total consumption. Another example is the retention rate is the ratio between the number of active users on the next day and the number of new users on the same day.

[0105] In this embodiment, the ratio indicator The first indicator is B, and the second indicator is A. The first average value of the first indicator is determined based on the multi-bucket comparison data of the first indicator: Determine the first variance of the first indicator based on the multi-bucket comparison data and the first average value of the first indicator Determine the second average value of the second indicator based on the multi-bucket comparison data of the second indicator: Determine the second variance of the second indicator based on the multi-bucket comparison data and the second average value of the second indicator Calculate the covariance s between the first indicator and the second indicator based on the multi-bucket control data, the first mean and the second mean ab .

[0106] The sample mean of the ratio indicator is the ratio between the second mean and the first mean. According to the sample mean, first mean, second mean, first variance, second variance and covariance of the ratio indicator, the sample variance of the ratio indicator can be determined:

[0107] In this embodiment, the sample mean and sample variance of the proportion indicator are determined based on multi-bucket comparison data of the proportion indicator, without the need for multiple resampling, thereby reducing the number of experiments and the amount of calculation required to calculate the business significance threshold, and improving the efficiency of calculating the business significance threshold.

[0108] Optionally, based on any of the above embodiments, a business significance threshold corresponding to the business indicator is determined based on the statistic and the confidence interval expression, including: substituting the statistic into the confidence interval expression to obtain the confidence interval corresponding to the indicator difference; and determining the business significance threshold corresponding to the business indicator based on the minimum and maximum values ​​of the confidence interval.

[0109] It should be noted that the asymptotic distribution of the indicator difference is: Taking the preset confidence level of 95% as an example, the confidence interval expression of the indicator difference is:

[0110] For the size indicator, the sample mean is Converges to the expected μ according to probability, sample variance s 2 Converges to the variance according to probability, that is According to the continuous mapping theorem, we get: Based on Slutsky's Theorem, we can determine the asymptotic distribution of the indicator difference: Then we can determine the confidence interval expression of the scale index as follows:

[0111] The sample size n and sample mean of the scale indicator are and the sample variance s 2 Substituting the confidence interval expression of the scale indicator into the confidence interval expression, the confidence interval of the scale indicator can be obtained, and the minimum and maximum values ​​of the confidence interval of the scale indicator are used as the business significance threshold of the scale indicator.

[0112] For ratio indicators, similarly, due to the consistency of the estimator, the sample mean converges to the expectation according to probability, and the sample variance converges to the variance according to probability; the asymptotic difference of the indicator is: Among them, the sample mean of the ratio index is The sample variance is The asymptotic distribution of the proportional index can be obtained: Then we can determine the confidence interval expression of the proportion index as follows:

[0113] The sample size n and sample mean of the proportion index are and the sample variance s 2 Substituting the confidence interval expression of the proportion indicator into the confidence interval expression, the confidence interval of the proportion indicator can be obtained, and the minimum and maximum values ​​of the confidence interval of the proportion indicator are used as the business significance threshold of the proportion indicator.

[0114] It should be noted that since the sample mean of the ratio indicator and the sample variance s 2 It is determined based on the sample mean and sample variance of the first indicator, the sample mean and sample variance of the second indicator, and the covariance between the first indicator and the second indicator. Substituting the sample mean and sample variance of the first indicator, the sample mean and sample variance of the second indicator, and the covariance between the first indicator and the second indicator into the confidence interval expression of the proportion indicator, the confidence interval of the proportion indicator can be obtained.

[0115] In this embodiment, the statistics of the business indicator are substituted into the confidence interval expression of the business indicator to obtain the confidence interval, and then the business significance threshold is determined, so that the business significance threshold can reflect the actual fluctuation range of the business indicator in the AA experiment, so that the business significance threshold can be used to determine whether the experimental results are caused by natural fluctuations or actual differences in the business, thereby improving the reliability of the business experiment.

[0116] Optional, such as Figure 2 As shown, the method for determining the business significance threshold includes:

[0117] S201, obtaining an asymptotic distribution of a random variable difference; the random variable difference represents a difference between a first random variable and a second random variable, and a variance of the asymptotic distribution is determined based on an expected mean and a variance of each of the first random variable and the second random variable;

[0118] S202, using first control data obtained by performing a first experiment on a business indicator of a first object as a first random variable; using second control data obtained by performing a second experiment on a business indicator of the first object as a second random variable; based on the asymptotic distribution of the difference of the random variables, when the expected mean and variance of the first control data are respectively equal to the expected mean and variance of the second control data, obtaining the asymptotic distribution of the indicator difference; the indicator difference represents the difference of the business indicator in the experiment of the first object;

[0119] S203, constructing a confidence interval expression of the indicator difference under a preset confidence level according to the asymptotic distribution of the indicator difference;

[0120] S204, obtaining control group data obtained by conducting a business indicator experiment on the first object; the control group data includes multiple buckets of control data; the number of buckets of the multiple buckets of control data is used as the sample size of the business indicator;

[0121] S205, when the business indicator includes a scale indicator, determining an average value of the multi-bucket control data, and using the average value as the sample mean of the scale indicator; determining a variance of the multi-bucket control data based on the multi-bucket control data and the average value, and using the variance as the sample variance of the scale indicator; the statistics include sample size, sample mean, and sample variance;

[0122] S206, the ratio index is the ratio between the first index and the second index. When the business index includes the ratio index, the first mean value and the first variance of the first index, the second mean value and the second variance of the second index, and the covariance between the first index and the second index are determined based on the multi-bucket comparison data; the sample mean of the ratio index is determined based on the first mean value and the second mean value; the sample variance of the ratio index is determined based on the first mean value, the sample mean, the first variance, the second variance and the covariance; the statistics include the sample size, the sample mean and the sample variance;

[0123] S207, substituting the statistic into the confidence interval expression to obtain the confidence interval corresponding to the indicator difference; determining the business significance threshold corresponding to the business indicator based on the minimum and maximum values ​​of the confidence interval; the business significance threshold is used to determine the statistical significance of the experimental results corresponding to the business indicator in the business experiment.

[0124] Next, the method for determining a service significance threshold provided by the embodiment of the present disclosure is compared with the related technology to illustrate the effect of the method for determining a service significance threshold provided by the embodiment of the present disclosure.

[0125] Obtain the application's disk data, calculate a hash value based on the identifier of the disk data, use the hash value as the SID (Session ID), construct 2000 SIDs, pair every two SIDs to obtain 1000 experiments, and calculate the business significance threshold at 95% confidence through 1000 experiments; since the business significance threshold is calculated based on a large number of experiments, the business significance threshold is used as the true value to compare the quality of the business significance threshold determination method provided by the embodiment of the present disclosure and the business significance threshold determined by the related technology.

[0126] The relevant technology can adopt the Bucket Bootstrap method. Bucket Bootstrap is optimized based on the Bootstrap method. Bootstrap is a resampling technology that generates multiple sample sets by sampling the experimental data with replacement, calculates statistics through multiple sample sets, and estimates distribution characteristics; Bucket Bootstrap divides the experimental data into multiple groups and samples each group, which reduces the redundancy of the data and reduces the storage and computing costs.

[0127] Figure 3-Figure 9 When the real business significance threshold (true value) is determined based on actual experiments, the probability density curve of each scale indicator is marked with a 95% confidence interval. The 95% confidence interval is the calculated business significance threshold. The horizontal axis of the probability density curve is the difference value of the experimental results, and the vertical axis is the probability density.

[0128] Figure 3 The probability density curve of the difference value of PV (Page View) is shown, and its 95% confidence interval is (-1.59, 1.47);

[0129] Figure 4 The probability density curve of the difference value of UV (Unique Visitor) is shown, and its 95% confidence interval is (-0.84, 0.88);

[0130] Figure 5 The probability density curve of the difference value of goal (target number) is shown, and its 95% confidence interval is (-1.56, 1.46);

[0131] Figure 6 The probability density curve of the difference value of activity is shown, and its 95% confidence interval is (-1.12, 1.13);

[0132] Figure 7 The probability density curve of the difference value of satisfied consumption is shown, and its 95% confidence interval is (-1.61, 1.55);

[0133] Figure 8 The probability density curve of the difference value of the first satisfied consumption is shown, and its 95% confidence interval is (-1.61, 1.55);

[0134] Fig. 9 The probability density curve of the difference value of satisfactory consumption PV is shown, and its 95% confidence interval is (-1.45, 1.4);

[0135] The comparison results on scale indicators are shown in Table 1.

[0136] Table 1

[0137]

[0138] Figure 10-Figure 16 When the real business significance threshold (true value) is determined based on actual experiments, the probability density curve of each proportion indicator is marked with a 95% confidence interval. The 95% confidence interval is the calculated business significance threshold. The horizontal axis of the probability density curve is the difference value of the experimental results, and the vertical axis is the probability density.

[0139] Fig.10 The probability density curve of the difference value of the satisfied consumption rate is shown, and its 95% confidence interval is (-0.95, 1.03);

[0140] Fig.11 The probability density curve of the difference value of the secondary retention rate is shown, and its 95% confidence interval is (-0.48, 0.48);

[0141] Fig.12 The probability density curve of the difference value of the proportion of point behavior is shown, and its 95% confidence interval is (-0.41, 0.43);

[0142] Fig.13 The probability density curve of the first satisfied consumption ratio is shown, and its 95% confidence interval is (-0.91, 0.92);

[0143] Fig.14The probability density curve of the difference value of the proportion of satisfied consumption is shown, and its 95% confidence interval is (-0.42, 0.41);

[0144] Fig.15 The probability density curve of the difference in average activity per capita is shown, and its 95% confidence interval is (-0.5, 0.51);

[0145] Fig.16 The probability density curve of the difference value of the query ratio is shown, and its 95% confidence interval is (-1.5, 1.48);

[0146] The comparison results on the proportion index are shown in Table 2, and the unit of the data in Table 2 is %.

[0147] Table 2

[0148]

[0149] According to the above comparison, it can be seen that in terms of scale index and proportion index, the business significance threshold determined by the embodiment of the present disclosure is closer to the true value of the business significance threshold than the business significance threshold determined by the Bucket Bootstrap method. Among them, in terms of scale index, the error of the Bucket Bootstrap method is 0.05%, and the error of the embodiment of the present disclosure is 0.03%. In terms of proportion index, the error of the Bucket Bootstrap method is 0.03%, and the error of the embodiment of the present disclosure is 0.02%.

[0150] It can be seen that the method for determining the business significance threshold provided by the embodiment of the present disclosure has a smaller error and improves the accuracy of the business significance threshold; in addition, the business significance threshold of the business indicator is different under different business scenarios, different traffic sizes and different experimental cycles. Therefore, the Bucket Bootstrap method needs to construct different experiments for different business scenarios, different traffic sizes and different experimental cycles, and each Bucket Bootstrap method requires multiple resampling, involving a large amount of calculation, resulting in high cost and low efficiency. The embodiment of the present disclosure calculates the business significance threshold through statistics and confidence interval expressions, with small amount of calculation and simple process, which effectively reduces costs and improves efficiency.

[0151] The method for determining a business significance threshold provided by the embodiment of the present disclosure obtains a confidence interval expression of an indicator difference, determines a statistic based on control data obtained by conducting an experiment on a business indicator of a first object, substitutes the statistic into the confidence interval expression, and obtains a business significance threshold corresponding to the business indicator; since the indicator difference represents the difference of the business indicator in the experiment of the first object, that is, the confidence interval expression of the indicator difference, can reflect the natural fluctuation range of the indicator difference, and then the business significance threshold determined by the statistic determined by the experiment and the confidence interval expression of the indicator difference can determine whether the experimental result is caused by natural fluctuation or actual difference in the business, that is, the business significance threshold can be used to determine the significance of the business indicator in the business, avoiding the situation where the experimental result is significant but there is no actual improvement in the business, thereby improving the reliability of the business experiment; in addition, by determining the statistic based on the control data obtained by the experiment on the first object, and calculating the business significance threshold based on the statistic and the confidence interval expression, less data is involved, and the amount of calculation is small, so the business significance threshold can be obtained quickly, thereby improving the efficiency of determining the business significance threshold.

[0152] Fig.17 A flow chart of a business experiment method provided by an embodiment of the present disclosure, such as Fig.17 As shown, the method includes:

[0153] Step 1701: Conduct a business indicator experiment on the first object to obtain control group data of the business indicator.

[0154] Step 1702: Conduct a business indicator experiment on the second object to obtain experimental group data of the business indicator; the second object is different from the first object in version.

[0155] The execution subject of the embodiment of the present disclosure may be a business experiment device, and the business experiment device may be coupled to a server; it can be understood that the business experiment device and the device for determining the business significance threshold may belong to the same experiment system.

[0156] Optionally, the first object is a first version product, and the second object is a second version product. In actual applications, the second object may be a version upgrade of the first object. The business experiment is used to test the performance of the first object and the second object in terms of business indicators to determine whether the second object has any business improvement compared to the first object.

[0157] The experiment of the business indicator on the first object may be a control group experiment in the business experiment; the experiment of the business indicator on the second object may be an experimental group experiment in the business experiment; the business experiment may be an AB experiment, in which users are randomly divided into a control group (Group A) and an experimental group (Group B), the control group uses the first object, and the experimental group uses the second object. When the users in the control group use the first object, the control group data of the business indicator is obtained, and when the users in the experimental group use the second object, the experimental group data of the business indicator is obtained.

[0158] Step 1703: Determine the experimental results corresponding to the business indicators based on the control group data and the experimental group data, and determine the business significance of the experimental results according to the business significance threshold; the business significance threshold is determined according to the above-mentioned method for determining the business significance threshold.

[0159] Among them, the experimental result can be the business indicator of the second object, which is the improvement value compared with the business indicator of the first object; the business significance includes that the experimental result is significant in business, or the experimental result is not significant in business; when the experimental result is significant in business, it means that the business indicator of the second object has actual benefits in business.

[0160] In this embodiment, the difference in business indicators is determined based on the control group data and the experimental group data, and a statistical analysis is performed on the difference in business indicators to determine the difference value of the business indicator between the first object and the second object, and the difference value is used as the experimental result of the business indicator; illustratively, the experimental result can indicate that the second object has an improvement of a% (difference value) in business indicators compared to the first object.

[0161] Obtain the business significance threshold determined by the statistic and confidence interval expression based on the control group data. The business significance threshold includes the minimum value and the maximum value. The process of calculating the business significance threshold can refer to the description of the above-mentioned method for determining the business significance threshold. After obtaining the business significance threshold, compare the difference value with the business significance threshold to obtain the business significance of the experimental results.

[0162] In a possible implementation, the business experiment method includes: obtaining a confidence interval expression of an indicator difference; the indicator difference represents the difference of the business indicator in an experiment on a first object; determining a statistic based on the control group data obtained from an experiment on the business indicator of the first object; determining a business significance threshold corresponding to the business indicator based on the statistic and the confidence interval expression; conducting an experiment on the business indicator of a second object to obtain experimental group data of the business indicator; the second object has a different version from the first object; determining the experimental results corresponding to the business indicator based on the control group data and the experimental group data, and determining the business significance of the experimental results based on the business significance threshold.

[0163] The business experiment method provided by the embodiment of the present disclosure obtains control group data of the business indicator by conducting an experiment on the business indicator of a first object, obtains experimental group data of the business indicator by conducting an experiment on the business indicator of a second object, determines the experimental results of the business indicator based on the control group data and the experimental group data, and determines the business significance of the experimental results through a business significance threshold. Since the business significance threshold can reflect the natural fluctuation range of the indicator difference, the business significance threshold can reflect whether the experimental results are caused by natural fluctuations of the experimental results or by actual differences in the business, and then determines the business significance of the experimental results through the business significance threshold, thereby avoiding the situation where the experimental results are significant but there is no actual improvement in the business, and improving the reliability of the business experiment.

[0164] Optionally, based on any of the above embodiments, the business significance threshold includes a first threshold and a second threshold; the experimental result includes a difference value of the business indicator between the first object and the second object; the business significance of the experimental result is determined based on the business significance threshold, including: when the difference value is between the first threshold and the second threshold, determining that the experimental result of the business indicator is not business significant; when the difference value does not fall between the first threshold and the second threshold, determining that the experimental result of the business indicator is business significant.

[0165] The difference value may reflect the degree of improvement in the business indicator of the second object compared to the first object; the business significance includes: the business is not significant or the business is significant.

[0166] In this embodiment, the minimum value included in the business significance threshold is used as the first threshold, and the maximum value included in the business significance threshold is used as the second threshold. If the difference value is between the first threshold and the second threshold, it means that the difference value may be caused by natural fluctuations, and the experimental result is that the business is not significant. If the difference value is not between the first threshold and the second threshold, it means that the difference value is not only caused by natural fluctuations, and the experimental result is that the business is significant.

[0167] In actual applications, multiple experiments were randomly constructed using the market data of the first object and the second object. Based on the control group data corresponding to the first object, the method of the embodiment of the present disclosure was used to calculate the business significance threshold of [-0.14%, +0.14%] at a confidence level of 95%. The control group data corresponding to the first object and the experimental group data corresponding to the second object were analyzed, and it was determined that the per capita activity (business indicator) of the experimental group data increased by +0.17%. According to the business significance threshold, it can be determined that the per capita activity is significant in the business; through actual verification, it is determined that the increase in user activity actually increases business revenue.

[0168] In this embodiment, since the business significance threshold can reflect the natural fluctuation range of the indicator difference, the business significance threshold can be used to reflect whether the experimental results are caused by natural fluctuations in the experimental results or by actual differences in the business. The business significance of the experimental results is then determined through the business significance threshold, thereby avoiding the situation where the experimental results are significant but there is no actual improvement in the business, thereby improving the reliability of the business experiment. By associating the differences in business indicators with business benefits through the business significance threshold, it is possible to intuitively evaluate whether the benefits obtained from the experiment are large enough and have practical significance, that is, converting statistical results into business language, which can help understand the value of the experimental results and ensure the effectiveness of subsequent decisions.

[0169] Fig.18 A schematic diagram of the structure of a device 180 for determining a service significance threshold provided in an embodiment of the present disclosure, such as Fig.18 As shown, the device 180 for determining a service significance threshold includes:

[0170] The acquisition unit 1801 is used to acquire a confidence interval expression of an indicator difference; the indicator difference represents a difference of a business indicator in an experiment of the first object;

[0171] A statistic determination unit 1802, configured to determine a statistic based on control group data obtained from an experiment on a business indicator of a first object;

[0172] The business significance threshold determination unit 1803 is used to determine the business significance threshold corresponding to the business indicator based on the statistic and the confidence interval expression.

[0173] Optionally, based on any of the above embodiments, the statistic determining unit 1802 includes:

[0174] The first experiment module is used to obtain control group data obtained by conducting a business indicator experiment on the first object; the control group data includes multiple buckets of control data;

[0175] The second experimental module is used to use the number of buckets of multi-bucket control data as the sample size of the business indicator;

[0176] The third experimental module is used to determine the sample mean and sample variance of business indicators based on multi-bucket control data; statistics include sample size, sample mean and sample variance.

[0177] Optionally, based on any of the above embodiments, the business indicator includes a scale indicator; and the third experimental module includes:

[0178] The first processing submodule is used to determine the average value of the multi-bucket control data and use the average value as the sample mean of the scale indicator;

[0179] The second processing submodule is used to determine the variance of the multi-bucket control data based on the multi-bucket control data and the average value, and use the variance as the sample variance of the scale indicator.

[0180] Optionally, based on any of the above embodiments, the business indicator includes a ratio indicator, which is a ratio between the first indicator and the second indicator; and the third experimental module includes:

[0181] A third processing submodule is used to determine a first mean value and a first variance of the first indicator, a second mean value and a second variance of the second indicator, and a covariance between the first indicator and the second indicator according to the multi-barrel comparison data;

[0182] A fourth processing submodule, used to determine a sample mean of the proportion indicator according to the first mean value and the second mean value;

[0183] The fifth processing submodule is used to determine the sample variance of the proportion indicator according to the first average value, the sample mean, the first variance, the second variance and the covariance.

[0184] Optionally, based on any of the above embodiments, the service significance threshold determination unit 1803 includes:

[0185] The confidence interval determination module is used to substitute the statistic into the confidence interval expression to obtain the confidence interval corresponding to the indicator difference;

[0186] The business significance threshold determination module is used to determine the business significance threshold corresponding to the business indicator based on the minimum and maximum values ​​of the confidence interval.

[0187] Optionally, based on any of the above embodiments, the device further includes:

[0188] A construction unit is used to determine the asymptotic distribution of the indicator difference; based on the asymptotic distribution of the indicator difference, a confidence interval expression of the indicator difference under a preset confidence level is constructed.

[0189] Optionally, based on any of the above embodiments, the construction unit includes:

[0190] An acquisition module, configured to acquire an asymptotic distribution of a random variable difference; the random variable difference represents a difference between a first random variable and a second random variable, and a variance of the asymptotic distribution is determined based on an expectation and a variance of each of the first random variable and the second random variable;

[0191] A first processing module, configured to use first control data obtained by performing a first experiment on a business indicator of a first object as a first random variable;

[0192] A second processing module, used to use second control data obtained from a second experiment on the business indicator of the first object as a second random variable;

[0193] The third processing module is used to obtain the asymptotic distribution of the indicator difference based on the asymptotic distribution of the random variable difference when the expectation and variance of the first control data are respectively equal to the expectation and variance of the second control data.

[0194] Fig.19 A schematic diagram of the structure of a business experiment device 190 provided in an embodiment of the present disclosure, such as Fig.19 As shown, the business experimental device 190 includes:

[0195] The first experimental unit 1901 is used to conduct an experiment on a business indicator on a first object to obtain control group data of the business indicator;

[0196] The second experiment unit 1902 is used to perform an experiment on a business indicator on a second object to obtain experimental group data of the business indicator; the second object is different from the first object in version;

[0197] The business significance determination unit 1903 is used to determine the experimental results corresponding to the business indicators based on the control group data and the experimental group data, and determine the business significance of the experimental results according to the business significance threshold; the business significance threshold is determined according to the above-mentioned business significance threshold determination method.

[0198] Optionally, based on any of the above embodiments, the business significance threshold includes a first threshold and a second threshold; the experimental result includes a difference value between the business indicator of the first object and the second object; and the business significance determination unit 1903 includes:

[0199] A first determination module, configured to determine that the experimental result of the business indicator is not significant when the difference value is between the first threshold and the second threshold;

[0200] The second determination module is used to determine that the experimental result of the business indicator is business significant when the difference value is not between the first threshold and the second threshold.

[0201] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0202] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0203] Fig. 20A schematic block diagram of an example electronic device 200 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0204] like Fig. 20 As shown, the device 200 includes a computing unit 2001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 2002 or a computer program loaded from a storage unit into a random access memory (RAM) 2003. In the RAM 2003, various programs and data required for the operation of the device 200 can also be stored. The computing unit 2001, the ROM 2002, and the RAM 2003 are connected to each other via a bus 2004. An input / output (I / O) interface 2005 is also connected to the bus 2004.

[0205] A number of components in the device 200 are connected to the I / O interface 2005, including: an input unit 2006, such as a keyboard, a mouse, etc.; an output unit 2007, such as various types of displays, speakers, etc.; a storage unit 2002, such as a disk, an optical disk, etc.; and a communication unit 2009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 2009 allows the device 200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0206] The computing unit 2001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 2001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 2001 performs the various methods and processes described above, such as a method for determining a business significant threshold and a business experiment method. For example, in some embodiments, the method for determining a business significant threshold and the business experiment method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 2002. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 200 via the ROM 2002 and / or the communication unit 2009. When the computer program is loaded into the RAM 2003 and executed by the computing unit 2001, one or more steps of the method for determining a business significant threshold and the business experiment method described above may be performed. Alternatively, in other embodiments, the computing unit 2001 may be configured to execute the method for determining the business significance threshold and the business experiment method in any other appropriate manner (for example, by means of firmware).

[0207] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0208] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0209] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0210] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0211] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0212] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0213] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0214] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining a business significance threshold, comprising: Obtaining a confidence interval expression of an indicator difference; the indicator difference represents a difference of a business indicator in an experiment of the first object; Determining a statistic based on control group data obtained from an experiment on the business indicator conducted on the first subject; Based on the statistic and the confidence interval expression, a business significance threshold corresponding to the business indicator is determined; the business significance threshold is used to determine the business significance of the experimental result corresponding to the business indicator in the business experiment.

2. The method according to claim 1, wherein: The determining of the statistic based on the control group data obtained by conducting the business indicator experiment on the first object includes: Acquire control group data obtained by conducting a business indicator experiment on the first object; the control group data includes multiple buckets of control data; The number of barrels of the multi-barrel comparison data is used as the sample size of the business indicator; The sample mean and sample variance of the business indicator are determined based on the multi-bucket comparison data; the statistic includes the sample size, the sample mean and the sample variance.

3. The method according to claim 2, wherein: The business indicators include scale indicators; Determining the sample mean and sample variance of the business indicator based on the multi-bucket comparison data includes: Determine an average value of the multi-barrel control data, and use the average value as the sample mean of the scale indicator; The variance of the multi-bucket comparison data is determined based on the multi-bucket comparison data and the average value, and the variance is used as the sample variance of the scale indicator.

4. The method according to claim 2, wherein: The business indicator includes a ratio indicator, and the ratio indicator is a ratio between the first indicator and the second indicator; Determining the sample mean and sample variance of the business indicator based on the multi-bucket comparison data includes: Determine a first mean value and a first variance of the first indicator, a second mean value and a second variance of the second indicator, and a covariance between the first indicator and the second indicator according to the multi-barrel comparison data; Determine a sample mean of the proportion indicator according to the first mean value and the second mean value; A sample variance of the proportion indicator is determined according to the first average value, the sample mean, the first variance, the second variance, and the covariance.

5. The method according to claim 1, wherein: The determining, based on the statistic and the confidence interval expression, a business significance threshold corresponding to the business indicator includes: Substituting the statistic into the confidence interval expression to obtain the confidence interval corresponding to the indicator difference; A business significance threshold corresponding to the business indicator is determined according to the minimum value and the maximum value of the confidence interval.

6. The method according to any one of claims 1 to 5, wherein: Before obtaining the confidence interval expression of the indicator difference, the method further includes: Determine the asymptotic distribution of indicator differences; According to the asymptotic distribution of the indicator difference, a confidence interval expression of the indicator difference under a preset confidence level is constructed.

7. The method according to claim 6, wherein: The step of determining the asymptotic distribution of the indicator difference comprises: Obtaining an asymptotic distribution of a random variable difference, wherein the random variable difference represents a difference between a first random variable and a second random variable, and the variance of the asymptotic distribution is determined based on the expectation and variance of each of the first random variable and the second random variable; Using first control data obtained from a first experiment on a business indicator of a first object as the first random variable; using second control data obtained by performing a second experiment on the business indicator on the first subject as the second random variable; Based on the asymptotic distribution of the random variable difference, the asymptotic distribution of the indicator difference is obtained when the expectation and variance of the first control data are respectively equal to the expectation and variance of the second control data.

8. A business experiment method, comprising: Conducting a business indicator experiment on the first subject to obtain control group data of the business indicator; Performing an experiment on the business indicator on a second object to obtain experimental group data of the business indicator; the second object is a different version from the first object; The experimental result corresponding to the business indicator is determined based on the control group data and the experimental group data, and the business significance of the experimental result is determined according to a business significance threshold; the business significance threshold is determined according to the method described in any one of claims 1-7.

9. The method according to claim 8, wherein: The business significance threshold includes a first threshold and a second threshold; the experimental result includes a difference value of the business indicator between the first object and the second object; Determining the business significance of the experimental result according to the business significance threshold includes: When the difference value is between the first threshold and the second threshold, determining that the experimental result of the service indicator is that the service is not significant; When the difference value is not between the first threshold and the second threshold, it is determined that the experimental result of the business indicator is business significant.

10. A device for determining a service significance threshold, comprising: Get unit, used to get the confidence interval expression of indicator difference; The indicator difference represents the difference of the business indicator in the experiment of the first object; A statistic determination unit, configured to determine a statistic based on control group data obtained by performing an experiment on the business indicator on the first object; A business significance threshold determination unit is used to determine the business significance threshold corresponding to the business indicator based on the statistic and the confidence interval expression.

11. The device according to claim 10, wherein: The statistic determination unit comprises: A first experiment module is used to obtain control group data obtained by conducting a business indicator experiment on the first object; the control group data includes multiple buckets of control data; The second experimental module is used to use the number of barrels of the multi-barrel control data as the sample size of the business indicator; The third experimental module is used to determine the sample mean and sample variance of the business indicator based on the multi-bucket control data; the statistics include the sample size, the sample mean and the sample variance.

12. The device according to claim 11, wherein The business indicator includes a scale indicator; the third experimental module includes: A first processing submodule, used for determining an average value of the multi-barrel control data, and using the average value as a sample mean of the scale indicator; The second processing submodule is used to determine the variance of the multi-bucket comparison data according to the multi-bucket comparison data and the average value, and use the variance as the sample variance of the scale indicator.

13. The device according to claim 11, wherein: The business indicator includes a ratio indicator, and the ratio indicator is a ratio between the first indicator and the second indicator; The third experimental module includes: A third processing submodule, configured to determine a first mean value and a first variance of the first indicator, a second mean value and a second variance of the second indicator, and a covariance between the first indicator and the second indicator according to the multi-barrel comparison data; a fourth processing submodule, configured to determine a sample mean of the proportion indicator according to the first mean value and the second mean value; The fifth processing submodule is used to determine the sample variance of the proportion indicator according to the first average value, the sample mean, the first variance, the second variance and the covariance.

14. The device according to claim 10, wherein: The business significance threshold determination unit includes: A confidence interval determination module, used for substituting the statistic into the confidence interval expression to obtain the confidence interval corresponding to the indicator difference; The business significance threshold determination module is used to determine the business significance threshold corresponding to the business indicator according to the minimum value and the maximum value of the confidence interval.

15. The device according to any one of claims 10 to 14, wherein: The device also includes: A construction unit is used to determine the asymptotic distribution of the indicator difference; and according to the asymptotic distribution of the indicator difference, a confidence interval expression of the indicator difference under a preset confidence level is constructed.

16. The device according to claim 15, wherein: The building block comprises: An acquisition module, configured to acquire an asymptotic distribution of a random variable difference; the random variable difference represents a difference between a first random variable and a second random variable, and a variance of the asymptotic distribution is determined based on an expectation and a variance of each of the first random variable and the second random variable; A first processing module, configured to use first control data obtained by performing a first experiment on a business indicator of a first object as the first random variable; A second processing module, configured to use second control data obtained by performing a second experiment on the business indicator on the first object as the second random variable; The third processing module is used to obtain the asymptotic distribution of the indicator difference based on the asymptotic distribution of the random variable difference when the expectation and variance of the first control data are respectively equal to the expectation and variance of the second control data.

17. A business experimental device, comprising: A first experimental unit is used to conduct a business indicator experiment on the first object to obtain control group data of the business indicator; A second experiment unit is used to perform an experiment on the business indicator on a second object to obtain experimental group data of the business indicator; the second object is different from the first object in version; A business significance determination unit, used to determine the experimental results corresponding to the business indicators based on the control group data and the experimental group data, and determine the business significance of the experimental results according to a business significance threshold; the business significance threshold is determined according to the method described in any one of claims 1-7.

18. The device according to claim 17, wherein: The business significance threshold includes a first threshold and a second threshold; the experimental result includes a difference value of the business indicator between the first object and the second object; the business significance determination unit includes: A first determination module, configured to determine, when the difference value is between the first threshold and the second threshold, that the experimental result of the business indicator is not significant; The second determination module is used to determine that the experimental result of the business indicator is business significant when the difference value is not between the first threshold and the second threshold.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images