Data processing method, medium, apparatus, and computing device

Abstract

Embodiments of the present disclosure provide a data processing method, medium, device and computing device, by determining first index data of control group users and experimental group users using a target application before the experiment starts and second index data after the experiment; after the experiment starts, the target application provides a preset function to the experimental group users; for any day after the experiment, a first difference is determined according to the second index data corresponding to the two groups of users respectively, the first difference is corrected according to the first index data corresponding to the two groups of users respectively, and a second difference is obtained; a hypothesis test judgment index corresponding to any day after the experiment is determined according to the second difference, so as to output indication information of whether to apply the preset function, by correcting the first difference after the experiment, variance reduction is realized, the sensitivity of the experiment detection is improved, and whether to online the preset function can be determined more efficiently and accurately, so that the online of the preset function meets the needs of customers, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of computer technology, and more specifically, the embodiments of this disclosure relate to data processing methods, media, apparatus, and computing devices. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] A / B testing is essentially a controlled experiment. In internet application scenarios, it specifically refers to an online controlled experiment. For example, a portion of the online users of a target application are randomly and evenly divided into different groups. Preset functions are provided to the application corresponding to the experimental group users, user behavior is recorded to form experimental data, and finally, the experimental effect is evaluated by statistical means. That is, by comparing a certain indicator data, it is determined whether the experimental effect is significant, thereby determining whether the preset function should be applied to the target application.

[0004] When determining whether the experimental effect is significant, the usual practice is to conduct a two-sample hypothesis test based on the sample indicators after the A / B experiment. Specifically, online users are randomly and evenly divided into experimental and control groups. After providing preset functions to the target application corresponding to the experimental group users, the sample indicators of the experimental and control groups are counted daily. The p-value is calculated based on the mean, variance and sample size of the two groups, and the p-value is used to determine whether the experimental effect is significant.

[0005] In the above methods, due to the randomness of online traffic diversion and the diversity of user characteristics, the sample index data inevitably have a large variance. According to the calculation method of the test statistic, when the denominator is large, that is, when the variance of the difference between the sample means of the experimental group and the control group after the experiment is large, the value of the test statistic is often small and the p value is often large. At this time, it is difficult to indicate that the experimental effect is significant, that is, the detection sensitivity is low. Summary of the Invention

[0006] This disclosure provides a data processing method, medium, apparatus, and computing device to correct the difference in sample means between the experimental group and the control group after an experiment, thereby reducing the variance corresponding to the corrected difference in sample means and improving the sensitivity of experimental detection.

[0007] In a first aspect of this disclosure, a data processing method is provided, the method comprising:

[0008] The control group and experimental group of users using the target application were identified. The first indicator data for each day of the two groups of users was obtained for N days before the start of the experiment and the second indicator data for each day of the M days after the start of the experiment. After the start of the experiment, the target application provided preset functions to the experimental group of users.

[0009] For any day after the experiment, the first difference is determined based on the second indicator data corresponding to the two groups of users, and the first difference is corrected based on the first indicator data corresponding to the two groups of users to obtain the second difference;

[0010] Based on the second difference, determine the hypothesis testing judgment index corresponding to any day after the experiment, and output the indication information on whether to apply the preset function on the target application based on the hypothesis testing judgment index M days after the experiment.

[0011] In a second aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the method provided in the first aspect.

[0012] In a third aspect of this disclosure, a data processing apparatus is provided, the apparatus comprising:

[0013] The user identification module is used to identify the control group users and experimental group users using the target application, and to obtain the first indicator data for each day of the two groups of users in the N days before the start of the experiment and the second indicator data for each day of the M days after the start of the experiment; wherein, after the start of the experiment, the target application provides preset functions to the experimental group users.

[0014] The correction module, for any day after the experiment, is used to determine the first difference based on the second indicator data corresponding to the two groups of users, and to correct the first difference based on the first indicator data corresponding to the two groups of users to obtain the second difference.

[0015] The output module is used to determine the hypothesis testing judgment index corresponding to any day after the experiment based on the second difference, and output an indication message on whether to apply the preset function on the target application based on the hypothesis testing judgment index M days after the experiment.

[0016] In a fourth aspect of this disclosure, a computing device is provided, comprising: at least one processor and a memory; the memory storing computer execution instructions; and at least one processor executing the computer execution instructions stored in the memory, such that at least one processor performs the method provided in the first aspect.

[0017] In this embodiment, the method involves determining the first indicator data before the experiment and the second indicator data after the experiment for both the control group and the experimental group using the target application. After the experiment begins, the target application provides preset functions to the experimental group. For any day after the experiment, a first difference is determined based on the second indicator data corresponding to the two groups of users. The first difference is then corrected based on the first indicator data corresponding to the two groups of users to obtain a second difference. A hypothesis testing judgment index is determined based on the second difference for any day after the experiment. An indication message is output based on the hypothesis testing judgment index indicating whether to apply the preset functions to the target application. By using the first indicator data before the experiment to correct the first difference after the experiment, the variance of the second difference is reduced compared to the variance of the first difference, thereby improving the sensitivity of the experimental detection. Furthermore, this method can more efficiently and accurately determine whether to provide preset functions to the target application, ensuring that the final launch of the preset functions meets customer needs and improves the user experience. Attached Figure Description

[0018] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0019] Figure 1 A schematic diagram illustrating an application scenario provided according to an embodiment of this disclosure is shown.

[0020] Figure 2 A schematic flowchart of a data processing method according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A schematic flowchart of another data processing method provided according to an embodiment of the present disclosure is shown.

[0022] Figure 4 This diagram illustrates a user backtracking data display and statistical calculation method according to an embodiment of the present disclosure.

[0023] Figure 5 An online experimental verification curve is schematically shown according to an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of the structure of a computer-readable storage medium provided according to an embodiment of the present disclosure is shown.

[0025] Figure 7 A schematic diagram of the structure of a data processing apparatus provided according to an embodiment of the present disclosure is shown.

[0026] Figure 8 A schematic diagram of the structure of a computing device provided according to an embodiment of the present disclosure is shown.

[0027] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0028] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0029] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0030] According to embodiments of this disclosure, a data processing method, medium, apparatus, and computing device are proposed.

[0031] In this document, it should be understood that the terminology used is for convenience of understanding only and does not imply any limitation on its meaning. Furthermore, any number of elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0032] In addition, the data involved in this disclosure may be data authorized by the user or fully authorized by all parties. The collection, dissemination and use of the data shall comply with the requirements of relevant national laws and regulations. The implementation methods / executives of this disclosure may be combined with each other.

[0033] The following is a description of the terminology used in this disclosure:

[0034] Metric data: Quantifiable records of user behavior, such as number of clicks and playback duration;

[0035] Sample vs. Population: In the scenario of conducting A / B testing on a target application, the population can be considered as the set of all users of the target application. A sample is formed by randomly selecting a portion of users from the population. The population statistic is generally difficult to obtain, and it is necessary to estimate the population statistic through the sample statistic. Due to the randomness of the sampling process, the sample statistic is also a random variable.

[0036] Confidence interval: Based on the sample mean, the confidence interval is the range within which the population mean is likely to be distributed with a 95% probability. The smaller the sample variance and the larger the sample size, the more compact the confidence interval, and the more reliable it is to use the sample to estimate the population.

[0037] Two-sample hypothesis testing involves inferring the statistical relationship between two populations represented by two sets of samples. For example, the H0 hypothesis states that the means of the indicators in the two populations are equal, while the alternative hypothesis states that the means of the indicators in the two populations are not equal. Two-sample hypothesis testing involves drawing two corresponding samples from the aforementioned populations and using the sample statistics to test whether the H0 hypothesis holds true at a given confidence level. In A / B experiments, it is used to determine whether the means of the indicators of the populations represented by the experimental group and the control group are equal, and thus to determine whether the experimental effect is significant.

[0038] Test statistic: A comprehensive statistic calculated based on the mean, variance, and sample size of the indicator data of two groups of samples. The larger the value of this comprehensive statistic, the more significant the difference between the means of the two groups of samples, and the more significant the experimental effect is considered.

[0039] p-value: The probability value calculated based on the test statistic. The smaller the p-value, the lower the probability that the H0 hypothesis is true, that is, the H0 hypothesis should be rejected. Invention Overview

[0041] The inventors have discovered that existing A / B tests, when performing two-sample hypothesis testing, directly calculate the test statistic based on the indicator data of the experimental group and the user group after the experiment, and then calculate the p-value to determine whether the experimental effect is significant. However, due to the randomness of online traffic distribution and the diversity of user characteristics, the sample indicator data inevitably have a large variance. If the variance is large, the value of the test statistic is often small, making it difficult to indicate that the experimental effect is significant, i.e., the detection sensitivity is not high.

[0042] Other existing technologies achieve variance reduction through stratified statistics. Stratified statistics refers to performing two-sample hypothesis testing based on post-experimental sample data. When calculating the sample statistic, the sample needs to be divided into several mutually exclusive and complete subclasses, such as users being divided according to dimensions like gender, new / old users, and commonly used device operating systems. The variance of each subclass is calculated and summed before calculating the test statistic and p-value. However, this method prioritizes variance reduction, and finding the most effective partitioning dimensions relies on human experience.

[0043] To address the aforementioned issues, this solution proposes a data processing method. Two-sample hypothesis testing based on post-experimental indicator data is a crucial evaluation method in A / B experiments for determining the significant effectiveness of experimental strategies. However, the randomness of user selection in the experimental and control groups leads to large variances in the sample indicators. For example, if the experimental group consists of young people and the control group consists of elderly people, the indicator data for the experimental group is generally higher than that for the control group before the application's preset functions are implemented, making it impossible to determine the significance of the preset functions. By correcting the pre-experimental data indicators, the influence of differences in user selection on the post-experimental indicator data is eliminated. This reduces the variance of the second difference (the corrected difference between the means of the experimental and control groups after the experiment) compared to the variance of the first difference (the uncorrected difference between the means of the experimental and control groups after the experiment). By modifying the data processing procedure based on the same data, the experimental effect becomes more significant, thereby improving the sensitivity of the experimental detection.

[0044] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0045] Application Scenarios Overview

[0046] First refer to Figure 1 As shown, Figure 1 The illustration schematically depicts an application scenario provided by an embodiment of this disclosure. The method is applied to a computing device, such as a server or terminal device, which can be a mobile phone, computer, tablet, or similar device. The computing device can interact with the user, acquire the user's click operations, and based on these click operations, obtain the corresponding indicator data of the experimental group and control group using the target application before and after the experiment. The method then corrects the indicator data of the two groups of users before the experiment, obtaining the difference in the corrected indicator data between the two groups. This allows for the determination of the corrected hypothesis testing judgment indicator, which in turn outputs indication information. When the above method is applied to a server, the server directly controls the target application according to the indication information, such as providing preset functions to the target application for all users. When the above method is applied to a terminal device, the terminal device can output the indication information to the server, which then provides preset functions to the target application for all users.

[0047] Exemplary methods

[0048] The following is combined with Figure 1 Application scenarios, refer to Figures 2-5This description outlines a data processing method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0049] refer to Figure 2 , Figure 2 A schematic flowchart illustrating a data processing method according to an embodiment of this disclosure is shown. Figure 2 As shown, this method is applied to a computing device, and the method includes:

[0050] Step S201: Determine the control group users and experimental group users using the target application, and obtain the first indicator data for each of the two groups of users for N days before the start of the experiment and the second indicator data for each of the M days after the experiment.

[0051] In one exemplary embodiment of this disclosure, after the experiment begins, the target application provides preset functions to the experimental group users.

[0052] When conducting A / B experiments, it is necessary to set up experimental and control groups. Different experimental strategies are applied to the experimental and control groups; for example, providing preset functions to users in the experimental group allows for the acquisition of indicator data for each group. For instance, in an online controlled experiment, we want to determine whether preset functions set in the target application will increase user click-through rates. For the control group, no preset functions are set in the target application; for the experimental group, the preset functions are set in the target application. We then analyze the indicator data of the experimental and control groups to determine whether the preset functions increase user click-through rates. If they do increase click-through rates, the preset functions can be extended to all users.

[0053] To obtain the corresponding indicator data for the experimental and control groups, online users can be randomly and evenly divided into two groups, designated as the experimental group and the control group, respectively. For example, a subset of users can be selected as the experimental group, and another subset as the control group.

[0054] The experiment is divided into two phases: the AA (Analog-Assessment) phase and the AB (Analog-Based Experiment) phase. In the AA phase, no experimental strategies are applied to either the experimental or control groups for N days. The daily sample size and primary metric data, such as clicks, are then recorded for both groups. The AA phase is essentially the phase before the experiment begins; for example, no preset functions are set on the target applications for either the experimental or control groups during this phase.

[0055] During the A / B testing phase, user groups remain unchanged; that is, the labels for the experimental or control groups remain constant. The experimental strategy is applied to users in the experimental group, while the control group is not treated as a control and is maintained for M days. The daily sample size and second metric data, such as clicks, are then recorded for both groups. The A / B testing phase is the phase after the experiment begins. For example, during the A / B testing phase, preset functions are set on the application for the experimental group, while no preset functions are set on the application for the control group.

[0056] Furthermore, the metrics data before and after the experiment can be inconsistent. For example, the metric data before the experiment could be the number of clicks, while the metric data after the experiment could be the number of plays; or, the metric data before the experiment could be the number of plays, while the metric data after the experiment could be the number of clicks.

[0057] For any user in the experimental or control group, if no corresponding indicator data is obtained for that user, that is, if the user does not use the target application on that day, the indicator data can be set to 0 or other constants.

[0058] Step S202: For any day after the experiment, determine the first difference based on the second indicator data corresponding to the two groups of users respectively, and correct the first difference based on the first indicator data corresponding to the two groups of users respectively to obtain the second difference.

[0059] After obtaining the first and second indicator data for the experimental and control groups, respectively, the significance of the experimental effect can be analyzed based on the first and second indicator data.

[0060] To facilitate the measurement of the effectiveness of providing preset functions to users, the differences in the corresponding indicator data of the experimental group and the control group after the experiment can be calculated, that is, the difference between the second indicator data of the experimental group and the second indicator data of the control group can be calculated.

[0061] Optionally, for any day after the experiment, for the experimental group, aggregate by user dimension and calculate the average of the second indicator data for each user. Similarly, for the control group, aggregate by user dimension and calculate the average of the second indicator data for each user. The difference between the average of the second indicator data for the experimental group and the average of the second indicator data for the control group is obtained as the first difference.

[0062] For the experimental and control groups, the randomness of user selection leads to randomness in the first and second indicator data, consequently resulting in randomness in the calculated first difference, meaning its value will fluctuate. The first difference reflects the effectiveness of preset functions set in the target application, but it contains a certain margin of error. Due to the randomness of grouping, users in the experimental group may correspond to the first demographic group, while users in the control group may correspond to the second. The first demographic group may inherently use the target application more frequently than the second, resulting in higher click volumes. In other words, there is a significant difference in click volumes between the two groups before the experiment. To eliminate the influence of user selection on the significance of the experiment, the first indicator data from the two groups before the experiment is used to correct the first difference, thus eliminating the impact of user selection differences on the post-experiment second indicator data.

[0063] Step S203: Determine the hypothesis testing judgment index corresponding to any day after the experiment based on the second difference, and output the indication information on whether to apply the preset function on the target application based on the hypothesis testing judgment index M days after the experiment.

[0064] For any day after the experiment, a hypothesis testing criterion can be calculated. When the hypothesis testing criterion meets a preset condition, it indicates that the experimental effect is significant based on the experimental data for that day. For example, the experimental effect can be determined to be significant when the hypothesis testing criterion for L days out of M days after the experiment meets the preset condition.

[0065] When the experimental results are determined to be significant, instructions can be directly output to apply the preset function to the target application, so that staff can apply the preset function to all users on the target application.

[0066] The data processing method provided by this disclosure determines the first indicator data of the control group users and the experimental group users before the start of the experiment and the second indicator data after the experiment. After the experiment starts, the target application provides preset functions to the experimental group users. For any day after the experiment, a first difference is determined based on the second indicator data corresponding to the two groups of users. The first difference is then corrected based on the first indicator data corresponding to the two groups of users to obtain a second difference. A hypothesis testing judgment index is determined based on the second difference for any day after the experiment. An indication message on whether to apply the preset functions to the target application is output based on the hypothesis testing judgment index. By using the first indicator data before the experiment to correct the first difference after the experiment, the variance of the second difference is reduced compared to the variance of the first difference, thereby improving the sensitivity of the experimental detection. In addition, this method can more efficiently and accurately determine whether to provide preset functions to the target application, so that the final launch of the preset functions can meet customer needs and improve the user experience.

[0067] refer to Figure 3 , Figure 3 A schematic flowchart of another data processing method provided according to an embodiment of this disclosure is shown. Figure 3 As shown, the method includes:

[0068] Step S301: Determine the control group users and experimental group users using the target application, and obtain the first indicator data for each of the two groups of users for N days before the start of the experiment and the second indicator data for each of the M days after the experiment.

[0069] In one exemplary embodiment of this disclosure, after the experiment begins, the target application provides preset functions to the experimental group users.

[0070] The above step S301 can be referred to Figure 2 Step S201 in the illustrated embodiment will not be repeated here.

[0071] For any day after the experiment, the corresponding second difference is calculated using the methods in steps S302 to S304.

[0072] Step S302: For any user in the experimental group and the control group, determine the first average value based on the N first indicator data corresponding to the user.

[0073] To correct for the initial differences using the initial indicator data from before the experiment, the average value of the initial indicator data for each user in both the experimental and control groups can be calculated for the N days prior to the experiment. In other words, for each user, the data is aggregated daily to obtain the initial average value.

[0074] Figure 4The illustration shows a schematic diagram of user backtracking data display and statistical calculation according to an embodiment of the present disclosure. For user 10001, the average value of the first indicator data over N days is calculated to obtain the corresponding avg1, which is the first average value.

[0075] Step S303: For the experimental group and the control group, determine the second average value based on the first average value corresponding to each user.

[0076] After determining the first average value for each user, the average value for the first N days for either the experimental or control group is calculated, which is essentially aggregation by user, to obtain the second average value. For example... Figure 4 As shown, the average of each first average is calculated to obtain avg2, which is the second average.

[0077] Step S304: Calculate the difference between the second average value corresponding to the experimental group and the second average value corresponding to the control group, and correct the first difference based on the difference to obtain the second difference.

[0078] After calculating the second average value for the experimental group and the second average value for the control group, the difference between the two is calculated. This difference reflects the difference in indicator data between the two groups due to user selection. When correcting for the first difference based on this difference, the second difference can be obtained by directly calculating the difference between the first difference and this difference, or by calculating the product of this difference and a fixed coefficient, and then calculating the difference between the first difference and this product to obtain the second difference.

[0079] By first calculating the first average and then calculating the second average based on the first average, the average values ​​of the indicator data of the experimental group and the control group before the experiment can be measured simply and conveniently, such as the average number of clicks of the experimental group. There is no need to obtain the indicator data of the corresponding number of days before the experiment based on the number of days after the experiment, making the handling of the experimental period more flexible, and the amount of historical data to be traced back is smaller, reducing resource consumption.

[0080] In one embodiment of this disclosure, correcting the first difference based on the difference to obtain a second difference includes:

[0081] The product of the correction coefficient and the difference is determined, and the difference between the first difference and the product is determined as the expression for the second difference.

[0082] The first difference is the difference between the third average values ​​of the second indicator data corresponding to the two groups of users.

[0083] When calculating the second difference, a correction coefficient can be set, and the correction coefficient can be multiplied by the difference between the second average value corresponding to the experimental group and the second average value corresponding to the control group. The difference between the first difference and the product is determined as the expression for the second difference.

[0084] The expression for the second difference is as follows: This represents the second indicator data corresponding to the experimental group users. This represents the second indicator data corresponding to the control group users. This represents the first indicator data corresponding to the control group users. This represents the first indicator data for the control group users; θ represents the correction coefficient.

[0085]

[0086] For any day after the experiment, the first difference can be calculated. For example, on the first day after the experiment, the first difference is obtained by subtracting the avg3 of the control group from the avg3 of the experimental group.

[0087] By setting correction coefficients, the correction coefficients can be solved, and the determined correction coefficients, compared to the set fixed values, can control the magnitude of the variance corresponding to the second difference, thereby improving the experimental results.

[0088] Step S305: Determine the expression corresponding to the variance of the second difference based on the expression of the second difference.

[0089] Step S306: When the variance of the second difference is minimized, determine the value of the correction coefficient for any day after the experiment.

[0090] Once the expression for the second difference is determined, the expression for its variance can be determined. The variance of the second difference is a quadratic function opening upwards. This variance has a minimum value; the value of the correction coefficient is determined when the variance of the second difference is at its minimum. When the variance of the second difference is at its minimum, the reduction in both variances compared to the variance corresponding to the first difference is the greatest.

[0091] After calculation, the value of the correction factor can be determined according to the following formula:

[0092]

[0093] in, This represents the covariance between the second indicator data corresponding to the experimental group users and the first average value corresponding to the N days prior to the experiment. Nt represents the covariance between the second indicator data corresponding to the control group users and the first mean data corresponding to the N days prior to the experiment; Nt represents the number of users in the experimental group; Nc represents the number of users in the control group. This represents the first variance determined based on the first average value of each user in the experimental group before the experiment. This represents the first variance determined based on the first average value corresponding to each user in the control group before the experiment.

[0094] In calculating the covariance of the index data before and after the experiment, the method used is to calculate the covariance of the average index data over multiple days before the experiment and the second index data for each day after the experiment. Alternatively, it can be to calculate the covariance of the first index data for each day before the experiment and the second index data for each day after the experiment, or it can be to calculate the covariance of the cumulative average index data over multiple days before the experiment and the second index data for multiple days after the experiment.

[0095] Because the calculation of the second difference takes into account the index data of both the experimental and control groups, the determined correction coefficient also considers the index data of both groups simultaneously. This ensures that the experimental effect is equivalent for both groups, avoiding a situation where the experimental group has the smallest variance while the control group does not. Furthermore, this value minimizes the variance of the second difference.

[0096] The covariance for any day after the experiment is determined based on the second indicator data and the first average value for each user in the experimental group and the control group, respectively. The number of users in the experimental group and the control group is determined, and the first variance is determined based on the first average value for each user in the experimental group and the control group. The covariance represents the relationship between the average value of the first indicator data before the experiment and the second indicator data of the users after the experiment. The value of the correction coefficient is determined based on the covariance, the number of users, and the first variance.

[0097] When acquiring Figure 4 After obtaining the first and second indicator data for each user, the aforementioned statistics can be calculated, such as the covariance and first variance of the two user groups. Once these statistics are determined, the value of the correction coefficient can be accurately obtained.

[0098] Step S307: Determine the hypothesis testing judgment index corresponding to any day after the experiment based on the value of the correction coefficient.

[0099] Once the correction coefficient is determined, the hypothesis testing indicator for that day can be obtained. This indicator can be either a p-value or a confidence interval. Based on the determined indicator, the significance of the experimental effect can be determined. For example, when the indicator is a p-value, a p-value less than 0.05 indicates a significant difference in the means of the two sets of data, suggesting a significant experimental effect.

[0100] By calculating the expression for the variance of the second difference, we can obtain the value of the correction coefficient when the variance of the second difference is minimized. This allows us to achieve the maximum reduction in variance, and the hypothesis testing judgment index determined based on the value of this correction coefficient is also more accurate.

[0101] In one embodiment of this disclosure, the hypothesis testing judgment index is the p-value; determining the hypothesis testing judgment index corresponding to any day after the experiment based on the value of the correction coefficient includes:

[0102] The second variance is determined based on the second indicator data corresponding to each user in the experimental group and the control group; the value of the test statistic is determined based on the value of the correction coefficient, the first variance, the second variance, the second mean, the third mean, the covariance, the number of users, and the expression of the test statistic; the expression of the test statistic is related to the expression of the second difference; the p-value is determined based on the value of the test statistic.

[0103] When calculating the p-value, the test statistic can be calculated first. For the corrected second difference, the formula for calculating the test statistic is as follows: where var(Δ′) represents the variance of the second difference.

[0104]

[0105] The formula for calculating var(Δ′) is as follows:

[0106]

[0107] Where, ρ 2 It is the square of the correlation coefficient after comprehensive processing, with a value between 0 and 1. It represents the correlation coefficient between the first indicator data before the experiment and the second indicator data after the experiment, determined by the experimental group and the control group. Its calculation formula is as follows:

[0108]

[0109] Based on the three formulas above, and the various statistics, the values ​​of the test statistics are determined. These statistics include: the second variance corresponding to the experimental group. The second variance corresponding to the control group Correction coefficient θ, first variance of the experimental group The first variance corresponding to the control group The second average value corresponding to the experimental group The second average value corresponding to the control group The third average value corresponding to the experimental group The third average value corresponding to the control group

[0110] Once the test statistic is determined, the p-value can be determined based on the test statistic. The formula for calculating the p-value is as follows:

[0111] p = 2(1-Φ) -1 (TS)

[0112] When the p-value is greater than 0.05, it indicates that the experimental effect is significant.

[0113] Once the expression for the second difference is determined, the expression for the test statistic can be determined based on the relationship between the test statistic and the second difference, thereby accurately determining the value of the test statistic based on the values ​​of each statistic.

[0114] In one embodiment of this disclosure, the hypothesis testing judgment index is a confidence interval; determining the hypothesis testing judgment index corresponding to any day after the experiment based on the value of the correction coefficient includes:

[0115] The confidence interval for the hypothesis test is determined based on the value of the correction coefficient, the first variance, the second variance, the second mean, the third mean, the covariance, the number of users, and the expression for the confidence interval; the expression for the confidence interval is related to the expression for the second difference.

[0116] When the confidence interval is used as the judgment index in hypothesis testing, it can be determined based on the expression for the confidence interval and the values ​​of each statistic. At a 95% confidence level, the population mean difference can be considered to be u. t -u c The confidence intervals for (the difference between the overall mean of the experimental group and the overall mean of the control group) are as follows:

[0117]

[0118] Once the confidence interval is determined, the value of the confidence interval can be used to determine whether the experimental effect is significant. When the lower limit of the confidence interval is greater than 0, it indicates positive significance; when the upper limit of the confidence interval is less than 0, it indicates negative significance. Negative significance means that the experimental effect deteriorates after adding the preset function.

[0119] Step S308: Based on the hypothesis test results M days after the experiment, output the indication information as to whether the preset function is applied on the target application.

[0120] Step S308 can be referred to Figure 2 The step S203 in the illustrated embodiment, which involves outputting an indication of whether to apply the preset function to the target application based on the hypothesis test index M days after the experiment, will not be described again here.

[0121] The following section corrects for the first difference by using the first indicator data before the experiment, and in principle, it is deduced that the variance has been reduced.

[0122] When the first difference is not corrected using the initial index data from before the experiment, the expression for the first difference is:

[0123]

[0124] The variance of the first difference is:

[0125]

[0126] By comparing the variance of the first difference var(Δ) and the variance of the second difference var(Δ′), it can be seen that the variance of the second difference is less than the variance of the first difference.

[0127] Furthermore, when the initial index data before the experiment is not used to correct for the first difference, the formula for calculating the test statistic is:

[0128]

[0129] Comparing the test statistic when correcting for the first difference using the first indicator data before the experiment with the test statistic when correcting for the first difference using the first indicator data before the experiment, we can see that the change in the corresponding test statistic after correction can be divided into two parts: changes in the numerator and the denominator. Regarding the numerator, the influence of population selection on the indicator data between the experimental and control groups before the experiment can be eliminated. Regarding the denominator, the variance is reduced, with a reduction magnitude of (1-ρ). 2 ).

[0130] Furthermore, obtain metric data from online experiments in real-world scenarios, such as through validation using a target application.

[0131] Figure 5 An online experimental verification curve is schematically illustrated according to an embodiment of the present disclosure, such as... Figure 5 As shown in the figure, from left to right, the p-values ​​corresponding to the test statistics of the differences between the five experimental groups and the control group are displayed. The horizontal axis represents the number of days after the experiment. The dashed line represents the p-value calculated by the method before correction, and the solid line represents the p-value calculated by the method after correction. As can be seen from the figure, the p-value calculated by the method after correction is lower than that calculated by the method before correction, and the trend of change is roughly the same. Moreover, some p-values ​​corresponding to the corrected scheme decrease to below 0.05, which improves the sensitivity of the experimental effect significance to a certain extent.

[0132] Exemplary media

[0133] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 6 The storage medium of the exemplary embodiments of this disclosure will be described.

[0134] refer to Figure 6 As shown, the storage medium 60 stores a program product for implementing the above-described method according to an embodiment of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes program code for causing a computing device to execute the data processing method provided in this disclosure. However, the program product of this disclosure is not limited thereto.

[0135] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0136] A readable signal medium may include data signals propagated in baseband or as part of a carrier wave, carrying program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0137] Program code for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0138] Exemplary device

[0139] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 7The data processing apparatus of the exemplary embodiments of this disclosure will be described to implement the method in any of the above data processing method embodiments. The implementation principle and technical effect are similar, and will not be repeated here.

[0140] The data processing apparatus 70 provided in this disclosure includes:

[0141] The user identification module 701 is used to identify the control group users and experimental group users using the target application, and to obtain the first indicator data for each day of the two groups of users in the N days before the start of the experiment and the second indicator data for each day of the M days after the start of the experiment; wherein, after the start of the experiment, the target application provides preset functions to the experimental group users.

[0142] The correction module 702, for any day after the experiment, is used to determine the first difference based on the second indicator data corresponding to the two groups of users respectively, and to correct the first difference based on the first indicator data corresponding to the two groups of users respectively, to obtain the second difference;

[0143] The output module 703 is used to determine the hypothesis testing judgment index corresponding to any day after the experiment based on the second difference, and output indication information on whether to apply the preset function on the target application based on the hypothesis testing judgment index M days after the experiment.

[0144] In one embodiment of this disclosure, the correction module 702 includes:

[0145] The first calculation unit is used to determine the first average value for any user in the experimental group and the control group based on the N first indicator data corresponding to the user.

[0146] The second calculation unit is used to determine a second average value for the experimental group and the control group, respectively, based on the first average value corresponding to each user.

[0147] The correction unit is used to calculate the difference between the second average value corresponding to the experimental group and the second average value corresponding to the control group, and correct the first difference based on the difference to obtain the second difference.

[0148] In one embodiment of this disclosure, when the correction unit corrects the first difference based on the difference to obtain the second difference, it is specifically used for:

[0149] The product of the correction coefficient and the difference is determined, and the difference between the first difference and the product is determined as the expression for the second difference; the first difference is the difference between the third average values ​​of the second indicator data corresponding to the two groups of users respectively.

[0150] In one embodiment of this disclosure, the output module 703 includes:

[0151] The first determining unit is used to determine the expression corresponding to the variance of the second difference based on the expression of the second difference;

[0152] The second determining unit is used to determine the value of the correction coefficient for any day after the experiment when the variance of the second difference is minimized.

[0153] The third determining unit is used to determine the hypothesis testing judgment index corresponding to any day after the experiment based on the value of the correction coefficient.

[0154] In one embodiment of this disclosure, the second determining unit is specifically used for:

[0155] The covariance for any day after the experiment is determined based on the second indicator data and the first average value for each user in the experimental group and the control group, respectively. The number of users in the experimental group and the control group is determined, and the first variance is determined based on the first average value for each user in the experimental group and the control group. The covariance represents the relationship between the average value of the first indicator data before the experiment and the second indicator data of the users after the experiment. The value of the correction coefficient is determined based on the covariance, the number of users, and the first variance.

[0156] In one embodiment of this disclosure, the hypothesis testing judgment index is the p-value; the third determining unit is specifically used for:

[0157] The second variance is determined based on the second indicator data corresponding to each user in the experimental group and the control group; the value of the test statistic is determined based on the value of the correction coefficient, the first variance, the second variance, the second mean, the third mean, the covariance, the number of users, and the expression of the test statistic; the expression of the test statistic is related to the expression of the second difference; the p-value is determined based on the value of the test statistic.

[0158] In one embodiment of this disclosure, the hypothesis testing judgment index is a confidence interval; the third determining unit is specifically used for:

[0159] The confidence interval for the hypothesis test is determined based on the value of the correction coefficient, the first variance, the second variance, the second mean, the third mean, the covariance, the number of users, and the expression for the confidence interval; the expression for the confidence interval is related to the expression for the second difference.

[0160] Exemplary computing device

[0161] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 8 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0162] Figure 8 The computing device 80 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0163] like Figure 8 As shown, the computing device 80 is presented in the form of a general-purpose computing device. The components of the computing device 80 may include, but are not limited to: at least one processing unit 801, at least one storage unit 802, and a bus 803 connecting different system components (including the processing unit 801 and the storage unit 802). The at least one storage unit 802 stores computer-executable instructions; the at least one processing unit 801 includes a processor that executes the computer-executable instructions to implement the methods described above.

[0164] The 803 bus includes a data bus, a control bus, and an address bus.

[0165] Storage unit 802 may include readable media in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory 8022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 8023.

[0166] Storage unit 802 may also include a program / utility 8025 having a set (at least one) program module 8024, such program module 8024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0167] The computing device 80 can also communicate with one or more external devices 804 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 805. Furthermore, the computing device 80 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 806. Figure 8 As shown, network adapter 806 communicates with other modules of computing device 80 via bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 80, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0168] It should be noted that although the data processing apparatus and several units / modules or sub-units / modules of the data processing apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0169] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0170] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: The control group and experimental group of users using the target application were identified. The first indicator data for each day of the two groups of users was obtained for N days before the start of the experiment and the second indicator data for each day of the M days after the start of the experiment. After the start of the experiment, the target application provided preset functions to the experimental group of users. For any day after the experiment, for the experimental group, aggregate according to the user dimension and calculate the average value of the second indicator data for each user. For the control group, aggregated by user dimension, and calculated the average value of the second indicator data for each user; The first difference is obtained by subtracting the average value of the second indicator data corresponding to the experimental group from the average value of the second indicator data corresponding to the control group. For any user in the experimental group and the control group, a first average value is determined based on the N first indicator data corresponding to the user. For the experimental group and the control group, a second average value is determined based on the first average value for each user. Calculate the difference between the second average value corresponding to the experimental group and the second average value corresponding to the control group, and correct the first difference based on the difference to obtain the second difference; Based on the second difference, determine the hypothesis testing judgment index corresponding to any day after the experiment, and output the indication information on whether to apply the preset function on the target application based on the hypothesis testing judgment index M days after the experiment.

2. The method according to claim 1, characterized in that, The first difference is corrected based on the difference to obtain the second difference, including: The product of the correction coefficient and the difference is determined, and the difference between the first difference and the product is determined as the expression for the second difference; the first difference is the difference between the third average values ​​of the second indicator data corresponding to the two groups of users respectively.

3. The method according to claim 2, characterized in that, Based on the second difference, determine the hypothesis testing criteria for any day after the experiment, including: Determine the expression corresponding to the variance of the second difference based on the expression for the second difference; When the variance of the second difference is minimized, determine the value of the correction coefficient for any day after the experiment. The hypothesis testing judgment index corresponding to any day after the experiment is determined based on the value of the correction coefficient.

4. The method according to claim 3, characterized in that, Determine the value of the correction factor for any day after the experiment, including: The covariance for any day after the experiment is determined based on the second indicator data and the first average value for each user in the experimental group and the control group, respectively. The number of users in the experimental group and the control group is determined, and the first variance is determined based on the first average value for each user in the experimental group and the control group. The covariance represents the relationship between the average value of the first indicator data before the experiment and the second indicator data of the users after the experiment. The value of the correction coefficient is determined based on the covariance, the number of users, and the first variance.

5. The method according to claim 4, characterized in that, The hypothesis testing judgment index is the p-value; the hypothesis testing judgment index corresponding to any day after the experiment is determined based on the value of the correction coefficient, including: The second variance is determined based on the second indicator data corresponding to each user in the experimental group and the control group; The value of the test statistic is determined based on the value of the correction coefficient, the first variance, the second variance, the second mean, the third mean, the covariance, the number of users, and the expression for the test statistic; the expression for the test statistic is related to the expression for the second difference. The p-value is determined based on the value of the test statistic.

6. The method according to claim 5, characterized in that, The hypothesis testing criterion is the confidence interval; The hypothesis testing criteria for any day after the experiment are determined based on the value of the correction coefficient, including: The confidence interval for the hypothesis test is determined based on the value of the correction coefficient, the first variance, the second variance, the second mean, the third mean, the covariance, the number of users, and the expression for the confidence interval; the expression for the confidence interval is related to the expression for the second difference.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 6.

8. A data processing apparatus, characterized in that, The device includes: The user identification module is used to identify the control group users and experimental group users using the target application, and to obtain the first indicator data for each day of the two groups of users in the N days before the start of the experiment and the second indicator data for each day of the M days after the start of the experiment; wherein, after the start of the experiment, the target application provides preset functions to the experimental group users. The correction module, for any day after the experiment, is used to aggregate and calculate the average value of the second indicator data for each user in the experimental group, and to aggregate and calculate the average value of the second indicator data for each user in the control group, and to obtain the first difference by subtracting the average value of the second indicator data for the experimental group from the average value of the second indicator data for the control group. The first difference is corrected based on the first indicator data corresponding to the two groups of users to obtain the second difference; The output module is used to determine the hypothesis testing judgment index corresponding to any day after the experiment based on the second difference, and output the indication information of whether to apply the preset function on the target application based on the hypothesis testing judgment index M days after the experiment. The correction module includes: The first calculation unit is used to determine the first average value for any user in the experimental group and the control group based on N first indicator data corresponding to the user. The second calculation unit is used to determine a second average value for the experimental group and the control group, respectively, based on the first average value corresponding to each user. The correction unit is used to calculate the difference between the second average value corresponding to the experimental group and the second average value corresponding to the control group, and correct the first difference based on the difference to obtain the second difference.

9. The apparatus according to claim 8, characterized in that, When the correction unit corrects the first difference based on the difference to obtain the second difference, it is specifically used for: The product of the correction coefficient and the difference is determined, and the difference between the first difference and the product is determined as the expression for the second difference; the first difference is the difference between the third average values ​​of the second indicator data corresponding to the two groups of users respectively.

10. The apparatus according to claim 9, characterized in that, The output module includes: The first determining unit is used to determine the expression corresponding to the variance of the second difference based on the expression of the second difference; The second determining unit is used to determine the value of the correction coefficient for any day after the experiment when the variance of the second difference is minimized. The third determining unit is used to determine the hypothesis testing judgment index corresponding to any day after the experiment based on the value of the correction coefficient.

11. The apparatus according to claim 10, characterized in that, The second determining unit is specifically used for: The covariance for any day after the experiment is determined based on the second indicator data and the first average value for each user in the experimental group and the control group, respectively. The number of users in the experimental group and the control group is determined, and the first variance is determined based on the first average value for each user in the experimental group and the control group. The covariance represents the relationship between the average value of the first indicator data before the experiment and the second indicator data of the users after the experiment. The value of the correction coefficient is determined based on the covariance, the number of users, and the first variance.

12. The apparatus according to claim 11, characterized in that, The hypothesis testing criterion is the p-value; the third determining unit is specifically used for: The second variance is determined based on the second indicator data corresponding to each user in the experimental group and the control group; The value of the test statistic is determined based on the value of the correction coefficient, the first variance, the second variance, the second mean, the third mean, the covariance, the number of users, and the expression for the test statistic; the expression for the test statistic is related to the expression for the second difference. The p-value is determined based on the value of the test statistic.

13. The apparatus according to claim 12, characterized in that, The hypothesis testing judgment index is a confidence interval; the third determining unit is specifically used for: The confidence interval for the hypothesis test is determined based on the value of the correction coefficient, the first variance, the second variance, the second mean, the third mean, the covariance, the number of users, and the expression for the confidence interval; the expression for the confidence interval is related to the expression for the second difference.

14. A computing device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1 to 6.

Images