AB experiment grouping flow distribution method and device and electronic equipment

Abstract

The invention provides an AB experiment grouping flow distribution method and device and electronic equipment, and relates to the technical field of computer application, and the method comprises the steps: obtaining the statistical data of an A group of experiment schemes and a B group of experiment schemes in an experiment process, and determining the tuning indexes of the A group of experiment schemes and the B group of experiment schemes based on the target data extracted from the statistical data, and calculating the confidence upper bounds of the group A and the group B based on the tuning index, and finally, comparing the confidence upper bounds of the group A and the group B to improve the distribution flow of the experiment scheme corresponding to the larger confidence upper bound. According to the scheme, the distribution flow of the A group experiment scheme and the B group experiment scheme is dynamically distributed according to the performance conditions of the A group experiment scheme and the B group experiment scheme, the distribution mode can adapt to changes of user behaviors and hobbies and interests in the experiment environment, more distribution flow is distributed for people with excellent performance, the experiment process of the excellent experiment scheme is accelerated, and the experiment efficiency is improved. A designer can be helped to quickly screen out an expected experiment scheme, and the experiment period is shortened.

Description

Technical Field

[0001] This invention relates to the field of computer application technology, and specifically to an AB experiment group flow allocation method, device, and electronic device. Background Technology

[0002] A / B testing is an experimental method based on the principle of scientific comparison. Its core is to show users two or more different product options, then collect and analyze the behavioral data of each user (such as click-through rate, conversion rate, dwell time, etc.) to evaluate the differences in the effectiveness of different product options and finally select the optimal option.

[0003] A / B testing, as an important means of evaluating the effectiveness of different product solutions and selecting the best solution based on scientific experimental methods, has now been widely used in many fields, including e-commerce, online advertising, user interface optimization, product function testing, and marketing strategy development.

[0004] In product development, A / B testing has become standard practice. Many companies have built mature testing platforms to provide strong support for automated testing and analysis. Meanwhile, with the continuous upgrading and iteration of machine learning and data analysis tools, the entire process of testing—from design and execution to result analysis—has become more precise and efficient. With continuous technological advancements, A / B testing can now handle complex multivariate testing and is suitable for more dynamic user environments. However, A / B testing suffers from long testing cycles, and changes in user behavior and interests in dynamic testing environments can affect the reliability of the results. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method, apparatus and electronic device for AB experiment group flow allocation, so as to improve the experimental efficiency of AB experiment and reduce the experimental cycle time.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A method for grouping flow allocation in an A / B experiment includes:

[0008] Obtain the corresponding statistical data for experimental schemes A and B in the AB experiment, and record them as the first statistical data and the second statistical data, respectively. The statistical data is obtained by statistically analyzing the recorded data generated when users respond to experimental schemes A and B.

[0009] Target data is extracted from the first statistical data and the second statistical data respectively through data crawling. The target data is used to determine the value of the optimization index, which is data used to characterize the quality of the experimental scheme.

[0010] Based on the extracted target data, determine the first optimization index for the experimental scheme of group A and the second optimization index for the experimental scheme of group B.

[0011] The first confidence upper bound corresponding to the experimental scheme of group A is calculated based on the first optimization index.

[0012] The second confidence upper bound corresponding to the experimental scheme of group B is calculated based on the second optimization index.

[0013] The allocation flow corresponding to the experimental scheme that increases the larger of the first confidence upper bound and the second confidence upper bound.

[0014] Optionally, in the above AB experiment grouping traffic allocation method, calculating the first confidence upper bound corresponding to the A group experimental scheme based on the first optimization index includes:

[0015] The total revenue of the A-group experimental scheme is calculated based on the first optimization index corresponding to the current period and the first optimization index corresponding to the historical period. The total revenue of the A-group experimental scheme is the sum of the first optimization indexes calculated for each period corresponding to the A-group experimental scheme.

[0016] The first confidence upper bound corresponding to the experimental scheme of group A is calculated based on the total benefit using a target algorithm. The target algorithm includes a confidence interval upper bound algorithm, a normal distribution method, a t-distribution method, or a nonparametric statistical method.

[0017] The second confidence upper bound corresponding to the experimental scheme of group B is calculated based on the second optimization index, including:

[0018] The total benefit of the B group experimental scheme is calculated based on the second optimization index collected in the current period and the second optimization index collected in the historical period. The total benefit of the B group experimental scheme is the sum of the second optimization indexes calculated in each period corresponding to the B group experimental scheme.

[0019] The second confidence upper bound corresponding to the experimental scheme of group B is calculated based on the total revenue using the confidence interval upper bound algorithm.

[0020] Optionally, in the above AB experiment grouping flow allocation method, when the target data contains only one data item,

[0021] Based on the extracted target data, the first optimization index for experimental group A and the second optimization index for experimental group B were determined, including:

[0022] The target data extracted from the first statistical data is used as the first optimization index;

[0023] The target data extracted from the second statistical data is used as the second optimization index.

[0024] Optionally, in the above AB experiment grouping flow allocation method, when the target data contains N data items, N is a positive integer not less than 2.

[0025] Based on the extracted target data, the first optimization index for experimental group A and the second optimization index for experimental group B were determined, including:

[0026] The first optimization index is calculated based on the target data extracted from the first statistical data and the weight coefficients corresponding to each target data.

[0027] The second optimization index is calculated based on the target data extracted from the second statistical data and the weight coefficients of each target data.

[0028] Optionally, in the above AB experiment group traffic allocation method, before extracting the target data from the first statistical data and the second statistical data through data scraping, the method further includes:

[0029] The type of promotional content for the A group experimental scheme and the B group experimental scheme is determined based on the type labeling of the promotional content. The promotional content includes at least one of the following: application software, commodities, film and television works, cultural and tourism works, and literary works.

[0030] Based on the target mapping relationship, target data that matches the promotional content and needs to be extracted from the first statistical data and the second statistical data are determined, wherein the target mapping relationship is used to store the correspondence between the promotional content and the target data to be extracted.

[0031] Optionally, in the above AB experiment group traffic allocation method, before extracting the target data from the first statistical data and the second statistical data through data scraping, the method further includes:

[0032] Obtain the target data selection instruction based on the operation of the input device;

[0033] Select the specific content contained in the target data based on the target data selection instruction.

[0034] Optionally, the above-mentioned AB experiment group flow allocation method also includes:

[0035] Determine whether the A / B experiment meets the termination conditions, including the experiment duration reaching the preset calibration duration, or the total benefit of the A group experimental scheme or the B group experimental scheme reaching the calibration benefit.

[0036] When the termination condition is met, the experiment ends and the experiment log is output. The experiment log includes at least the optimization indicators, total revenue and allocated traffic for the experimental scheme of group A or group B at each time point.

[0037] An AB experiment group flow distribution device includes:

[0038] The statistical data acquisition unit is used to acquire the corresponding statistical data of experimental scheme A and experimental scheme B in the AB experiment, which are respectively denoted as the first statistical data and the second statistical data. The statistical data is obtained by statistically analyzing the recorded data generated when users respond to experimental scheme A and experimental scheme B.

[0039] The optimization index calculation unit is used to extract target data from the first statistical data and the second statistical data through data crawling, respectively. The target data is used to determine the value of the optimization index, which is data used to characterize the quality of the experimental scheme. Based on the extracted target data, the unit determines the first optimization index corresponding to the experimental scheme of group A and the second optimization index corresponding to the experimental scheme of group B.

[0040] The confidence upper bound calculation unit is used to calculate the first confidence upper bound corresponding to the experimental scheme of group A based on the first optimization index; and to calculate the second confidence upper bound corresponding to the experimental scheme of group B based on the second optimization index.

[0041] The flow allocation unit is used to allocate flow for experimental schemes that increase the larger of the first confidence upper bound and the second confidence upper bound.

[0042] An electronic device includes at least one processing device and a storage device connected to the processing device, wherein:

[0043] The storage device is used to store computer programs;

[0044] The processing device is used to execute the computer program so that the electronic device can implement any of the above-described AB experiment group flow allocation methods.

[0045] A computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement any of the AB experiment grouping traffic allocation methods described above.

[0046] Based on the above technical solution, the AB experimental group traffic allocation method, apparatus, and electronic device provided in this embodiment of the invention first acquire statistical data generated when users respond to experimental schemes A and B during the experiment; secondly, based on the target data extracted from the statistical data, the optimization indexes for experimental schemes A and B are determined; thirdly, the upper confidence bounds for experimental schemes A and B are calculated based on the optimization indexes; and finally, by comparing the upper confidence bounds of experimental schemes A and B, the traffic allocation for the experimental scheme with the larger upper confidence bound is increased. In this solution, for group A... The traffic allocation for Experimental Plan B and Experimental Plan B is not fixed, but dynamically allocated based on their performance. This allocation method can adapt to changes in user behavior and interests in the experimental environment, allocating more traffic to high-performing plans and accelerating the experimental process of superior plans. This helps designers quickly select the desired experimental plans, reduces the experimental cycle, and because the amount of traffic required for each experimental plan changes indirectly based on the collected statistical data, the traffic allocation can adapt to changes in user behavior and interests, making the experimental results more reliable and accurate. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 A flowchart illustrating an AB experiment group flow allocation method provided in this application embodiment;

[0049] Figure 2 A flowchart illustrating an AB experiment group flow allocation method provided in another embodiment of this application;

[0050] Figure 3 A schematic diagram of the structure of an AB experiment group flow distribution device provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] In this embodiment, during the A / B test, the performance of each experiment is determined based on the feedback of each experimental data. The allocation of traffic to each experiment is dynamically adjusted based on the degree of excellence of each experiment, allocating more traffic to experiments with excellent performance, so as to quickly identify experimental schemes with better performance or high significance, thereby reducing the experimental cycle length and improving the reliability of experimental results.

[0054] See Figure 1 The AB experiment group flow allocation method disclosed in this application includes:

[0055] Step S101: Obtain statistical data of experimental schemes A and B in experiment AB;

[0056] The statistical data is obtained by collecting the recorded data generated by the target object (which can be the user targeted by the experimental scheme) in response to the experimental scheme A and the experimental scheme B in each collection period. For example, this recorded data may include the data generated when the target object clicks, browses, downloads or purchases the promotional content of the experimental scheme when responding to the experimental scheme A and the experimental scheme B. For example, these statistical data may include the number of clicks, the amount of rewards, the sales volume, the browsing time, the number of downloads, the browsing depth, etc.

[0057] During the A / B experiment, the specific content of the statistical data corresponding to each experimental scheme can be set in advance according to the experimental requirements. For example, one or more of the following data can be selected as the required statistical data based on the promotion content of the experimental scheme: number of clicks, amount of rewards, browsing time, number of downloads, browsing depth, sales volume, etc. Then, based on these data, the response data of each target object to the experimental scheme A and experimental scheme B are monitored, the required statistical data are extracted, and statistical data matching the experimental scheme A and experimental scheme B are constructed. In order to distinguish the statistical data of the experimental scheme A and experimental scheme B, the statistical data corresponding to the experimental scheme A is recorded as the first statistical data, and the statistical data corresponding to the experimental scheme B is recorded as the second statistical data.

[0058] Step S102: Extract target data from the first statistical data and the second statistical data respectively through data crawling. The target data is used to determine the value of the optimization index, which is data used to characterize the quality of the experimental scheme.

[0059] Designers can select and determine the specific content of the target data based on the evaluation criteria and design requirements. All data items included in the target data can be extracted from the statistical data through data scraping. That is, this step can extract the first target data from the first statistical data based on the data content of the target data, and extract the second target data from the second statistical data based on the data content of the target data.

[0060] In this scheme, the optimization metric refers to data used to characterize the quality of an experimental scheme. The specific content of this optimization metric can be set according to the design expectations. For example, if the expectation is to obtain an experimental scheme with more user clicks, then click count can be used as the optimization metric; if the expectation is to obtain an experimental scheme with longer user browsing time, then browsing time can be used as the optimization metric; if the expectation is to obtain more donations, then donation amount can be used as the optimization metric. In short, the specific content of the optimization metric differs under different expected goals, and the target data is the data used to determine the magnitude of the optimization metric. This application can pre-establish a correspondence between the extracted target data and the optimization metric. Through this correspondence, the optimization metric matching the extracted target data can be mapped.

[0061] Step S103: Based on the extracted target data, determine the first optimization index corresponding to the experimental scheme of group A and the second optimization index corresponding to the experimental scheme of group B.

[0062] In this step, to distinguish the optimization indicators of the experimental schemes in Group A and Group B, this application designates the optimization indicator of the experimental scheme in Group A as the first optimization indicator and the optimization indicator of the experimental scheme in Group B as the second optimization indicator. After obtaining the target data extracted from the first and second statistical data, the first optimization indicator corresponding to the experimental scheme in Group A and the second optimization indicator corresponding to the experimental scheme in Group B are determined based on the extracted target data. The first and second optimization indicators can be used to evaluate the superiority of the experimental schemes in Group A and Group B within the current collection period. The better the optimization indicator, the better the corresponding experimental scheme. For example, if the first optimization indicator is better than the second optimization indicator, it is considered that the performance of the experimental scheme in Group A is better than that of the experimental scheme in Group B in the current period; if the second optimization indicator is better than the first optimization indicator, it is considered that the performance of the experimental scheme in Group B is better than that of the experimental scheme in Group A in the current period.

[0063] Step S104: Calculate the first confidence upper bound corresponding to the experimental scheme of group A based on the first optimization index, and calculate the second confidence upper bound corresponding to the experimental scheme of group B based on the second optimization index.

[0064] The upper confidence bound (UCB) is a key indicator in statistics used to quantify the uncertainty of experimental results. In this scheme, to ensure the reliability of the analysis results, the overall performance of the experimental schemes A and B during the experimental process is further evaluated by the magnitude of the upper confidence bound (UCB) of the experimental schemes A and B. The larger the upper confidence bound (UCB), the better the performance of the corresponding experimental scheme. Therefore, in order to determine the performance of the experimental schemes A and B, this application can pre-calculate the upper confidence bounds of the experimental schemes A and B based on the first and second optimization indices using existing algorithms. The upper confidence bound corresponding to the experimental scheme A is recorded as the first upper confidence bound, and the upper confidence bound corresponding to the experimental scheme B is recorded as the second upper confidence bound.

[0065] Step S105: Increase the allocated flow corresponding to the experimental scheme that is the larger of the first confidence upper bound and the second confidence upper bound.

[0066] In this scheme, the superior experimental scheme between group A and group B is determined by comparing the magnitudes of the first and second confidence upper bounds. Specifically, the experimental scheme corresponding to the larger confidence upper bound is considered the superior scheme, and its allocated flow is increased. Since the total allocated flow is fixed, while increasing the allocated flow of the superior scheme, the allocated flow of the other experimental scheme in the A / B experiment needs to be decreased. The amount of flow increased or decreased each time can be fixed or determined based on the difference between the confidence upper bounds of group A and group B. For example, if the first confidence upper bound is greater than the second confidence upper bound, the flow of group A is increased by a preset step size; if the first confidence upper bound is less than the second confidence upper bound, the flow of group B is increased by a preset step size; if the first confidence upper bound is equal to the second confidence upper bound, the ratio of the allocated flow of group A and group B remains unchanged. For example, in a specific instance, the second optimization metric of the B group experimental scheme improved by 1% compared to the first optimization metric of the A group experimental scheme. The confidence upper bound was then calculated based on the total revenue of the A and B group experimental schemes. The magnitudes of the confidence upper bounds for the A and B group experimental schemes were compared, and the traffic was increased to the experimental scheme with the higher confidence upper bound. For instance, when the confidence upper bound of the B group experimental scheme was greater than that of the A group experimental scheme, and the ratio was 18:7, the allocated traffic for the B group experimental scheme was increased from 50% to 72%, and the allocated traffic for the A group experimental scheme was decreased to 28%.

[0067] As described above, the A / B experiment disclosed in this application first acquires statistical data generated during the experiment when users respond to experimental schemes A and B. Secondly, based on the statistical data, optimization indicators for experimental schemes A and B are determined. Thirdly, confidence upper bounds for experimental schemes A and B are calculated based on the optimization indicators. Finally, by comparing the confidence upper bounds of experimental schemes A and B, the allocated traffic for the experimental scheme with the larger confidence upper bound is increased. In this scheme, the allocated traffic for experimental schemes A and B is not fixed but dynamically allocated based on their performance. This allocation method can adapt to changes in user behavior and interests in the experimental environment, allocating more traffic to high-performing schemes and accelerating the experimental process of superior schemes. This helps designers quickly select the desired experimental schemes, reducing the experimental cycle. Furthermore, since the amount of traffic required for each experimental scheme changes indirectly based on the collected statistical data, the traffic allocation can adapt to changes in user behavior and interests, making the experimental results more reliable and accurate.

[0068] In this embodiment, when acquiring statistical data for experimental schemes A and B in the AB experiment, the data can be acquired based on a set collection period. The length of the collection period can be set according to user needs. For example, the sampling period can be any time interval such as 5 minutes, 1 hour, or 2 hours. Furthermore, the duration of the period can be a fixed duration, or it can be continuously increased as the experiment progresses. This is because, in the early stages of the experiment, the comparison results of the corresponding confidence upper bounds of experimental schemes A and B may fluctuate significantly. For example, within a certain time period, the confidence upper bound of experimental scheme A may be greater than that of experimental scheme B, while after a short period of time, the confidence upper bound of experimental scheme B may be greater than that of experimental scheme A. Therefore, in order to detect changes in the comparison results of the confidence upper bounds in a timely manner, the duration of the collection period set in the early stages of the experiment can be relatively short. As the experiment continues to grow, the statistical results corresponding to experimental schemes A and B accumulate, causing the comparison results of the confidence upper bounds of the two experimental schemes to tend to stabilize. Therefore, the duration of the collection period can be appropriately extended. Of course, the duration of the collection period in each time period can also be allocated based on world time. For example, the collection period is shorter when users are active (e.g., from 8:00 to 23:00) and longer when users are inactive (e.g., from 23:00 to 8:00 the next day). Configuring the collection period based on user activity can more accurately and effectively utilize the computing resources of the system using this method and prevent the ineffective use of computing resources.

[0069] In this embodiment, the application can use existing algorithms to calculate the upper confidence bounds of experimental schemes A and B. These existing algorithms can be any one of the following: the Confidence Interval Upper Bound Algorithm (UCB algorithm), the normal distribution method, the t-distribution method, or the nonparametric statistical method. Other algorithms are also possible. In this case, the total return of experimental scheme A can be calculated based on the first optimization index corresponding to the current period and the first optimization index corresponding to each historical period. The total return of experimental scheme A is the sum of the first optimization indices calculated for each period corresponding to experimental scheme A. Then, the target algorithm is used to calculate the first upper confidence bound of experimental scheme A based on the total return. Similarly, the total return of experimental scheme B is calculated based on the second optimization index corresponding to the current period and the second optimization indices collected for each historical period. The total return of experimental scheme B is the sum of the second optimization indices calculated for each period corresponding to experimental scheme B. Then, the target algorithm is used to calculate the second upper confidence bound of experimental scheme B based on the total return.

[0070] Taking the confidence interval upper bound algorithm as an example, the confidence interval upper bound algorithm can be based on the formula The upper confidence bound (UCB) for each experimental group is calculated. In this formula, S is the sum of the corresponding optimization metrics collected in each period, i.e., the total revenue; s is the standard deviation of the optimization metrics collected in each period; z is the quantile of the standard normal distribution; and n is the flow rate allocated to each experimental group. Generally, experimental groups with higher upper confidence bounds (UCBs) are more superior. This scheme prioritizes allocating flow resources to these experimental groups. This flow allocation method can quickly amplify the advantages of experimental groups with proven superior performance (avoiding resource waste on inferior groups) and actively explore potentially underestimated high-potential groups (preventing premature convergence to local optima). This dynamic adjustment mechanism ensures that resources always flow to experimental schemes that are "currently the best performing" or "potentially the best in the future," thereby minimizing the revenue gap with the optimal strategy under limited resources (i.e., minimizing regret) and ultimately maximizing long-term cumulative revenue. Furthermore, this solution also discloses the specific implementation process for calculating the first confidence upper bound corresponding to the experimental scheme of group A based on the first optimization index, and the second confidence upper bound corresponding to the experimental scheme of group B based on the second optimization index. For details, please refer to [link to relevant documentation]. Figure 2 This application also discloses the specific process for calculating the first and second confidence upper bounds based on the confidence interval upper bound algorithm. Specifically, the process includes:

[0071] Step S201: Calculate the total revenue of experimental scheme A based on the first optimization index collected in the current period and the first optimization index collected in the historical period; calculate the total revenue S of experimental scheme B based on the second optimization index collected in the current period and the second optimization index collected in the historical period.

[0072] In this step, during the A / B experiment, the first optimization index corresponding to each period is summed to obtain the total benefit of the A group experimental scheme during the experiment, and the second optimization index corresponding to each period is summed to obtain the total benefit of the B group experimental scheme during the experiment.

[0073] Step S202: Calculate the standard deviation s of the optimization index corresponding to each period of the experimental schemes A and B during the experiment.

[0074] In this step, after obtaining the optimization indices for each period of the experimental schemes A and B, the standard deviation s of the optimization indices for the experimental schemes A and B can be calculated based on mathematical formulas.

[0075] Suppose we have a set of optimization metrics for each period of an experiment: 5, 7, 9, 11, 13. First, we calculate the mean of these metrics, which is 9. Then, based on the mean of 9, we calculate the variance of these metrics, which is 10. Taking the square root of the variance gives us the standard deviation, which is 3.16. Using this process, we can calculate the standard deviations of the optimization metrics for experimental schemes A and B.

[0076] Step S203: Obtain the allocated flow n of experimental scheme A and experimental scheme B in the current period.

[0077] This step allows us to directly obtain the allocated traffic n for experimental schemes A and B in the current period based on the traffic allocation data.

[0078] Step S204: Substitute the total revenue S, standard deviation s, and allocated flow n corresponding to experimental schemes A and B into the formula respectively. The upper confidence bounds UCB for experimental schemes A and B are calculated.

[0079] In this embodiment, the target data may include only one data item, such as click-through rate (CTR) or download rate. In this case, the correspondence between the target data and the optimization metric can be: the target data equals the optimization metric. Therefore, the first target data captured from the first statistical data can be directly used as the first optimization metric, and the second target data captured from the second statistical data can be used as the second optimization metric. For example, when the target data only includes CTR, the CTR corresponding to experimental scheme A in the current period is extracted from the first statistical data and used as the first optimization metric; the CTR corresponding to experimental scheme B in the current period is extracted from the second statistical data and used as the second optimization metric.

[0080] In this embodiment, when a user needs to evaluate the merits of experimental schemes A and B based on N dimensions, the target data can include N data items, where N is a positive integer not less than 2. The calculation methods for the first and second optimization indicators include: calculating the first optimization indicator based on the target data items extracted from the first statistical data and their corresponding weight coefficients; and calculating the second optimization indicator based on the target data items extracted from the second statistical data and their weight coefficients. For example, the target data can simultaneously include click-through rate and download rate. When designing the correspondence between the target data and the optimization indicators, designers can pre-set the weight values ​​corresponding to each target data item. After capturing N target data items, each target data item is multiplied by its corresponding weight value, and then summed to calculate the corresponding optimization indicator. For example, the target data in this scheme can simultaneously include click-through rate (CTR) and download rate. After obtaining the first target data from the first statistical data, the product of the CTR and its corresponding weight coefficient in the first target data is calculated, as well as the product of the download rate and its corresponding weight coefficient. The sum of these products is then used as the first optimization metric. Similarly, after obtaining the second target data from the second statistical data, the product of the CTR and its corresponding weight coefficient in the second target data is calculated, as well as the product of the download rate and its corresponding weight coefficient. The sum of these products is then used as the second optimization metric. Therefore, this implementation can evaluate the merits of each experimental scheme from multiple dimensions, thereby selecting the optimal experimental scheme.

[0081] In this embodiment, considering that different content promotions focus on different target data, in order to improve the experimental results and meet user expectations, the specific content of the target data can be automatically configured according to the type of promotional content in the A-group and B-group experimental schemes. The type of promotional content in the A-group and B-group experimental schemes can be determined based on the type labeling of the promotional content. In this scheme, before conducting the A / B experiment, the type of promotional content can be pre-labeled. The type of promotional content includes at least one of the following: application software, commodities, film and television works, cultural and tourism works, and literary works. After determining the type of promotional content in the A-group and B-group experimental schemes based on the type labeling of the promotional content, the specific content of the target data matching the promotional content is determined based on the target mapping relationship. The target data is then extracted from the first statistical data and the second statistical data based on the specific content of the target data. The target mapping relationship is used to store the correspondence between the promotional content and the target data to be extracted. The content of the target data matched by different types of promotional content can be selected according to needs, that is, the correspondence between the two can be configured by the user.

[0082] In this embodiment, the user can also manually select the content of the target data to be extracted. Therefore, before extracting the target data from the first statistical data and the second statistical data through data scraping, the method further includes: obtaining a target data selection instruction generated based on the operation of an input device; and selecting one or more data contents included in the target data based on the target data selection instruction. The input device can be a mouse, keyboard, voice input device, etc. The user selects various contents in the statistical data by operating the mouse, keyboard, or voice input device, and uses the selected data as the included data of the target data. Then, based on this, data scraping is performed on the first statistical data and the second statistical data.

[0083] In the technical solution disclosed in this embodiment, the user can pre-set one or more termination conditions to control the termination of the experiment. The specific content of the termination condition can be set according to the design requirements. For example, it can be that the experiment duration reaches a certain duration, or the total revenue of a certain experimental scheme reaches the target revenue. At this time, during the experiment, it is determined whether the A and B experiments meet the termination conditions. The termination conditions include the experiment duration reaching the pre-set target duration, or the total revenue of the A group experimental scheme or the B group experimental scheme reaching the target revenue. When the A and B experiments meet the termination conditions, the experiment ends and the experiment log is output. The experiment log includes at least the statistical data, optimization indicators, total revenue, and allocated traffic size of the A group experimental scheme or the B group experimental scheme at each time node, so that the user can promptly identify the experimental scheme with excellent overall performance. At the same time, the designer can also analyze the performance of the A group experimental scheme or the B group experimental scheme based on the experiment log and make corresponding improvements.

[0084] This embodiment discloses an AB experiment group flow distribution device. For the specific working content of each unit in the device, please refer to the above method embodiment.

[0085] The AB experiment group flow allocation device provided in the embodiments of the present invention is described below. The AB experiment group flow allocation device described below can be referred to in correspondence with the AB experiment group flow allocation method described above.

[0086] See Figure 3 The AB experiment group flow allocation device disclosed in this embodiment may include: a statistical data acquisition unit 10, an optimization index calculation unit 20, a confidence upper bound calculation unit 30, and a flow allocation unit 40.

[0087] The statistical data acquisition unit 10 is used to acquire the corresponding statistical data of the A group experimental scheme and the B group experimental scheme in the AB experiment, which are respectively denoted as the first statistical data and the second statistical data. The statistical data is obtained by statistically analyzing the recorded data generated when the user responds to the A group experimental scheme and the B group experimental scheme.

[0088] The optimization index calculation unit 20 is used to extract target data from the first statistical data and the second statistical data through data crawling, respectively. The target data is used to determine the value of the optimization index, which is data used to characterize the quality of the experimental scheme. Based on the extracted target data, the first optimization index corresponding to the experimental scheme of group A and the second optimization index corresponding to the experimental scheme of group B are determined.

[0089] The confidence upper bound calculation unit 30 is used to calculate the first confidence upper bound corresponding to the experimental scheme of group A based on the first optimization index; and to calculate the second confidence upper bound corresponding to the experimental scheme of group B based on the second optimization index.

[0090] The flow allocation unit 40 is used to increase the allocated flow corresponding to the experimental scheme that increases the larger of the first confidence upper bound and the second confidence upper bound.

[0091] This embodiment discloses an electronic device, which may include at least one processing device and a storage device connected to the processing device, wherein: the storage device is used to store a computer program; the processing device is used to execute the computer program so that the electronic device can implement the AB experiment grouping traffic allocation method described above.

[0092] This application also provides a schematic diagram of the specific structure of an electronic device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as laptops, desktop computers, automation platforms, etc. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0093] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0094] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0095] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the AB experiment group traffic allocation methods provided in this application.

[0096] This application also provides a computer-readable storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the AB experiment group traffic allocation methods provided in this application.

[0097] The user information (including but not limited to user click-through rate, download rate, reward rate, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0098] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, in implementing this invention, the functions of each module can be implemented in one or more software and / or hardware components.

[0099] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0100] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0101] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0102] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0103] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for grouping flow distribution in an AB experiment, characterized in that, include: Obtain the corresponding statistical data for experimental schemes A and B in the AB experiment, and record them as the first statistical data and the second statistical data, respectively. The statistical data is obtained by statistically analyzing the recorded data generated when users respond to experimental schemes A and B. Target data is extracted from the first statistical data and the second statistical data respectively through data crawling. The target data is used to determine the value of the optimization index, which is data used to characterize the quality of the experimental scheme. Based on the extracted target data, determine the first optimization index for the experimental scheme of group A and the second optimization index for the experimental scheme of group B. The first confidence upper bound corresponding to the experimental scheme of group A is calculated based on the first optimization index. The second confidence upper bound corresponding to the experimental scheme of group B is calculated based on the second optimization index. The allocation flow corresponding to the experimental scheme that increases the larger of the first confidence upper bound and the second confidence upper bound.

2. The AB experiment group flow allocation method according to claim 1, characterized in that, The first confidence upper bound corresponding to the experimental scheme of group A is calculated based on the first optimization index, including: The total revenue of the A-group experimental scheme is calculated based on the first optimization index corresponding to the current period and the first optimization index corresponding to the historical period. The total revenue of the A-group experimental scheme is the sum of the first optimization indexes calculated for each period corresponding to the A-group experimental scheme. The first confidence upper bound for the experimental schemes in group A is calculated based on the total revenue using the objective algorithm. The second confidence upper bound corresponding to the experimental scheme of group B is calculated based on the second optimization index, including: The total revenue of the B group experimental scheme is calculated based on the second optimization index corresponding to the current period and the second optimization index collected in the historical period. The total revenue of the B group experimental scheme is the sum of the second optimization indexes calculated for each period corresponding to the B group experimental scheme. The second confidence upper bound corresponding to the experimental scheme of group B is calculated based on the total revenue using the objective algorithm.

3. The AB experiment group flow allocation method according to claim 1, characterized in that, When the target data contains only one data point, the first optimization index corresponding to the experimental scheme of group A and the second optimization index corresponding to the experimental scheme of group B are determined based on the extracted target data, including: The target data extracted from the first statistical data is used as the first optimization index; The target data extracted from the second statistical data is used as the second optimization index.

4. The AB experiment group flow allocation method according to claim 1, characterized in that, When the target data contains N data items, N is a positive integer not less than 2. Based on the extracted target data, the first optimization index for experimental group A and the second optimization index for experimental group B were determined, including: The first optimization index is calculated based on the target data extracted from the first statistical data and the weight coefficients corresponding to each target data. The second optimization index is calculated based on the target data extracted from the second statistical data and the weight coefficients of each target data.

5. The AB experiment group flow allocation method according to any one of claims 1-4, characterized in that, Before extracting the target data from the first statistical data and the second statistical data respectively through data scraping, the process also includes: The type of promotional content for the A group experimental scheme and the B group experimental scheme is determined based on the type labeling of the promotional content. Based on the target mapping relationship, determine the specific content of the target data that matches the promotional content; The target mapping relationship is used to store the correspondence between promotional content and the target data to be extracted.

6. The AB experiment group flow allocation method according to any one of claims 1-4, characterized in that, Before extracting the target data from the first statistical data and the second statistical data respectively through data scraping, the process also includes: Obtain the target data selection instruction based on the operation of the input device; Select the specific content contained in the target data based on the target data selection instruction.

7. The AB experiment group flow allocation method according to any one of claims 1-4, characterized in that, The method also includes: Determine whether the A / B experiment meets the termination conditions, including the experiment duration reaching the preset calibration duration, or the total benefit of the A group experimental scheme or the B group experimental scheme reaching the calibration benefit. When the termination condition is met, the experiment ends and the experiment log is output. The experiment log includes at least the optimization indicators, total revenue and allocated traffic for the experimental scheme of group A or group B at each time point.

8. An AB experiment group flow distribution device, characterized in that, include: The statistical data acquisition unit is used to acquire the corresponding statistical data of experimental scheme A and experimental scheme B in the AB experiment, which are respectively denoted as the first statistical data and the second statistical data. The statistical data is obtained by statistically analyzing the recorded data generated when users respond to experimental scheme A and experimental scheme B. The optimization index calculation unit is used to extract target data from the first statistical data and the second statistical data through data crawling, respectively. The target data is used to determine the value of the optimization index, which is data used to characterize the quality of the experimental scheme. Based on the extracted target data, the unit determines the first optimization index corresponding to the experimental scheme of group A and the second optimization index corresponding to the experimental scheme of group B. The confidence upper bound calculation unit is used to calculate the first confidence upper bound corresponding to the experimental scheme of group A based on the first optimization index. The second confidence upper bound corresponding to the experimental scheme of group B is calculated based on the second optimization index. The flow allocation unit is used to allocate flow for experimental schemes that increase the larger of the first confidence upper bound and the second confidence upper bound.

9. An electronic device, characterized in that, It includes at least one processing device and a storage device connected to the processing device, wherein: The storage device is used to store computer programs; The processing device is used to execute the computer program so that the electronic device can implement the AB experiment group flow allocation method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the AB experiment grouping traffic allocation method as described in any one of claims 1 to 7.

Images