Flow control method for A / B experiment, electronic equipment and storage medium
By monitoring and dynamically adjusting traffic allocation in A/B tests, using deep Q network and Bayesian optimization algorithms, the problems of unbalanced traffic allocation and insufficient control accuracy are solved, and the credibility and efficiency of experimental results are improved. Achieve refined flow control and stability.
Patent Information
- Application Number
- CN202411035044.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-06
AI Technical Summary
The existing A/B tests have uneven flow distribution, low control accuracy, and lack of real-time feedback mechanisms, resulting in poor experimental results deviations and stability, affecting the accuracy of strategy selection.
By setting up the experimental group and monitoring the access data in real time, using the deep Q network and Bayesian optimization algorithm to dynamically adjust the traffic ratio, combined with real-time data processing tools for refined flow control and real-time feedback, ensuring traffic balance and accurate control between experimental groups.
It improves the credibility and efficiency of experimental results, realizes precise control of flow and experimental stability, and can adjust the flow distribution strategy in a timely manner to deal with different situations, improving the accuracy of effect evaluation.
Smart Images

Figure CN119094448A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of software testing technology, and in particular to a flow control method, electronic device and storage medium for an A / B experiment. Background Art
[0002] A / B testing is an emerging web page optimization method that develops two plans for the same goal, allowing some users to use Plan A and others to use Plan B, and recording user usage to see which plan is more in line with the design. In today's digital and data-driven era, Internet companies widely use A / B testing to evaluate the effectiveness of different strategies, designs, or functions.
[0003] When conducting A / B testing, traffic allocation and control are crucial to the accuracy and credibility of experimental results. Common existing solutions are: one is to statically allocate traffic ratios, that is, determine in advance the proportion of traffic received by each group; however, this method cannot flexibly adjust dynamically according to real-time conditions, which may lead to problems such as uneven traffic and deviation in experimental results. Another is simple randomization control, that is, randomly allocating user traffic to different experimental groups; however, this method may cause large traffic fluctuations and poor stability, and cannot be finely controlled according to experimental needs.
[0004] In addition, existing technologies usually lack an effective real-time feedback mechanism and cannot monitor traffic usage and adjust allocation strategies in a timely manner, resulting in an inability to flexibly respond to different situations during the experiment. The above problems such as unbalanced traffic, low control accuracy, and poor real-time performance directly lead to poor test results and affect the final strategy selection judgment. Summary of the invention
[0005] In order to help improve the problems of unbalanced traffic and inaccurate control in existing A / B tests, the present application provides a traffic control method, electronic device and storage medium for an A / B experiment.
[0006] In a first aspect, the present application provides a flow control method for an A / B experiment, which adopts the following technical solution:
[0007] S10: Setting a first experimental group, a second experimental group and a control group, and allocating user access traffic according to a preset ratio, wherein the user access traffic is a preset total number of user accesses;
[0008] S20: Send the product strategy to be tested to the first experimental group for execution, monitor the access data and cross-validation results in real time, and output successful product strategies as feedback in real time;
[0009] S30: sending the user access traffic and successful product strategies of the first experimental group to the second experimental group for execution;
[0010] S40: Compare the user access results of the control group and the second experimental group in real time, and calculate the gain data brought by the successful product strategy.
[0011] By adopting the above technical solution, the flow rate can be accurately controlled, effectively improving the experimental efficiency and the reliability of the results.
[0012] In a second aspect, the present application provides an electronic device, which adopts the following technical solution:
[0013] An electronic device, comprising:
[0014] at least one processor;
[0015] Memory;
[0016] At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute a traffic control method for an A / B experiment.
[0017] In a third aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:
[0018] A computer-readable storage medium stores a computer program, which, when executed in a computer, causes the computer to execute any one of the A / B experiment flow control methods provided in the first aspect.
[0019] In summary, the technical solution of the present invention controls the flow based on real-time feedback data to ensure flow balance and accurate control between experimental groups. It realizes refined flow control, as well as real-time monitoring and adjustment of flow distribution, to improve control accuracy and experimental stability. By using real-time feedback data for flow control, the flow distribution strategy can be adjusted in time during the experiment, which improves the credibility of the experimental results and the accuracy of the effect evaluation. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 It is a flow chart of a flow control method for an A / B experiment provided in an embodiment of the present application;
[0021] Figure 2 It is a schematic diagram of a flow control method for an A / B experiment provided in an embodiment of the present application;
[0022] Figure 3 It is a module schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-3It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0024] The embodiment of the present application provides a flow control method for A / B experiment.
[0025] This embodiment uses the A / B experiment of strategy selection as an application example to specifically illustrate the above method. In actual implementation, the above method can also be used for other A / B experiments on design or function selection, which is not limited in this embodiment.
[0026] Please refer to the attached Figure 1 And attached Figure 2 As shown, this embodiment proposes a flow control method for an A / B experiment, and the method includes:
[0027] S10: Setting a first experimental group, a second experimental group and a control group, wherein the first experimental group, the second experimental group and the control group run the product strategy to be tested; adjusting the traffic ratio in real time according to the performance of different experimental groups through the DQN algorithm (Deep Q-Network) to obtain a preset proportional relationship; allocating user access traffic according to the preset proportional relationship, wherein the user access traffic is the preset total number of user visits;
[0028] S20: Send the product strategy to be tested to the first experimental group for execution, monitor the access data and cross-validation results in real time, and output successful product strategies as feedback in real time;
[0029] S30: Send the user access traffic and successful product strategies of the first experimental group to the second experimental group for operation; ensure the representativeness and balance of the experimental group.
[0030] S40: Compare the user access results of the control group and the second experimental group in real time, monitor and calculate the gain data brought by the successful product strategy in real time. For example, the gain data may be: the increase in recharge conversion rate, the growth of VIP subscriptions, the increase in average user income, etc.
[0031] Further, flow control is iteratively adjusted based on real-time data to optimize experimental results and continuously improve revenue growth strategies.
[0032] The solution can overcome the problems of traffic imbalance, low control accuracy, and lack of real-time feedback in traditional methods, thereby improving the efficiency and credibility of A / B experiments.
[0033] The technical solution of the present invention controls the flow based on real-time feedback data to ensure flow balance and accurate control between experimental groups. It realizes refined flow control, and monitors and adjusts flow distribution in real time to improve control accuracy and experimental stability. By using real-time feedback data for flow control, the flow distribution strategy can be adjusted in time during the experiment, which improves the credibility of the experimental results and the accuracy of effect evaluation.
[0034] On the basis of this embodiment, when the first experimental group runs the product strategy to be tested, cross-validation is performed through the multi-armed bandit algorithm, and the running effect is evaluated through real-time feedback data to output a successful product strategy.
[0035] It should be noted that the Multi-Armed Bandit (MAB) algorithm is used to solve the exploration and exploitation problem by selecting the slot machine arm to maximize the long-term reward. The algorithm selection depends on the problem and the goal, and the application scenarios are wide, such as marketing, recommendation, advertising, etc. Related software tools can assist algorithm experiments and applications.
[0036] In an embodiment of the present invention, the flow control method of the A / B experiment further includes: continuously iteratively optimizing the product strategy through a Bayesian optimization algorithm to improve the gain data brought by the successful product strategy.
[0037] It should be noted that Bayesian Optimization (BO) is an intelligent optimization method that combines Bayes' theorem and statistical learning theory to solve optimization problems. The optimization problem is the process of finding the optimal value of one or more parameters under limited computing resources. This method is widely used in machine learning, data mining, finance, bioinformatics and other fields.
[0038] On this basis, further, the user access behavior data is obtained through real-time data processing tools to transmit and monitor the traffic conditions of the experiment in real time. In this embodiment, the real-time data processing tool can be a tool such as Apache Kafka.
[0039] Furthermore, when cross-validating the effect: through a multiple validation mechanism, multiple cross-validations are performed using different sample sets or time periods. This method can reduce the impact of accidental factors and improve the reliability of the results.
[0040] Among them, an adaptive strategy is used to adjust the experimental parameters, and the flow ratio of each experimental group is adjusted according to the dynamic feedback information, so as to stabilize the experimental environment and ensure the consistency of the results.
[0041] Based on this embodiment, the flow control method of the A / B experiment also includes: calculating and analyzing the confidence interval, hypothesis probability (P-value), and relative value of the experimental results to evaluate the effectiveness of the product strategy. By analyzing the experimental results, the effectiveness of the product strategy is further evaluated and the error is reduced, thereby improving the stability of the experiment.
[0042] In the embodiment of the present invention, the static allocation ratio cannot dynamically adjust the flow distribution according to the real-time data, resulting in the problem of uneven flow between experimental groups. The present invention provides a flow control algorithm based on real-time feedback data, which can dynamically adjust the flow distribution ratio to ensure flow balance and accurate control between experimental groups.
[0043] Simple random control can easily cause large flow fluctuations and poor stability, and cannot precisely control the flow ratio of each experimental group. The present invention combines advanced algorithms and real-time data processing technology to achieve refined flow control, as well as real-time monitoring and adjustment of flow distribution, thereby improving control accuracy and experimental stability.
[0044] The lack of an effective real-time feedback mechanism makes the existing technology unable to respond to changing experimental requirements in a timely manner, limiting the flexibility and accuracy of the experimental process. The present invention uses real-time feedback data for flow control, which can adjust the flow distribution strategy in time during the experiment, thereby improving the credibility of the experimental results and the accuracy of the effect evaluation.
[0045] The existing technology has problems such as low accuracy of flow control and low credibility of experimental results due to limitations of control algorithms and data processing technology. The present invention adopts advanced algorithms and real-time data analysis technology to provide accurate flow control, which helps to improve the accuracy and credibility of experimental results and effectively solves the shortcomings of the existing technology in this regard.
[0046] In summary, this solution improves the field of real-time data processing and experimental design, fully utilizes real-time feedback data, and through advanced algorithms and precise control, makes up for many shortcomings of existing technologies in flow rate control and improves the efficiency and accuracy of A / B experiments.
[0047] The present application also provides an electronic device, such as Figure 3 As shown, Figure 3 The electronic device 700 shown includes: a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, such as through a bus 702. Optionally, the electronic device 700 may also include a transceiver 704. It should be noted that in actual applications, the transceiver 704 is not limited to one, and the structure of the electronic device 700 does not constitute a limitation on the embodiments of the present application.
[0048] Processor 701 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 701 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0049] The bus 702 may include a path to transmit information between the above components. The bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 702 may be divided into an address bus, a data bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0050] The memory 703 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.
[0051] The memory 703 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 701. The processor 701 is used to execute the application code stored in the memory 703 to implement the contents shown in the above method embodiment.
[0052] The electronic devices include, but are not limited to, mobile terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), etc., and fixed terminals such as digital TVs, desktop computers, etc. It can also be a server, etc. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0053] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the flow control method of the A / B experiment provided in the above embodiment.
[0054] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise clearly stated in this document, the execution of these steps is not strictly limited in order and can be performed in other orders.
[0055] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A flow control method for an A / B experiment, characterized in that: The method comprises: S10: Setting a first experimental group, a second experimental group and a control group, and allocating user access traffic according to a preset ratio, wherein the user access traffic is a preset total number of user accesses; S20: Send the product strategy to be tested to the first experimental group for execution, monitor the access data and cross-validation results in real time, and output successful product strategies as feedback in real time; S30: sending the user access traffic and successful product strategies of the first experimental group to the second experimental group for execution; S40: Compare the user access results of the control group and the second experimental group in real time, and calculate the gain data brought by the successful product strategy.
2. The flow control method for A / B experiment according to claim 1, characterized in that: When the first experimental group runs the product strategy to be tested, cross-validation is performed through the multi-armed bandit algorithm, and the operation effect is evaluated through real-time feedback data to output the successful product strategy.
3. The flow control method for A / B experiment according to claim 1, characterized in that: Before step S10, the process also includes: the first experimental group, the second experimental group and the control group run the product strategy to be tested, and adjust the traffic ratio in real time according to the performance of different experimental groups through the DQN algorithm to obtain a preset ratio relationship.
4. The flow control method for A / B experiment according to claim 1, characterized in that: Also includes: Continuously iterate and optimize product strategies through Bayesian optimization algorithms to improve the gain data brought by successful product strategies.
5. The flow control method for A / B experiment according to claim 1, characterized in that: Real-time data processing tools are used to obtain user access behavior data to transmit and monitor experimental traffic in real time.
6. The flow control method for A / B experiment according to claim 1, characterized in that: When cross-validation is effective: Through the multiple validation mechanism, multiple cross-validations are performed using different sample sets or time periods.
7. The flow control method for A / B experiment according to claim 1, characterized in that: Adaptive strategies are used to adjust experimental parameters, and the flow ratio of each experimental group is adjusted according to dynamic feedback information to ensure the consistency of results.
8. The flow control method for A / B experiment according to claim 1, characterized in that: Also includes: Calculate and analyze the confidence interval, hypothesis probability, and relative value of the experimental results to evaluate the effectiveness of the product strategy.
9. An electronic device, characterized in that: The electronic device includes: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the traffic control method for the A / B experiment as described in any one of claims 1 to 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the flow control method for the A / B experiment according to any one of claims 1 to 8.
