Abtest shunting method capable of overlapping, layering and barreling mechanism

Abstract

The invention provides an Abtest shunting method of an overlapping hierarchical bucket mechanism. The Abtest shunting method comprises the following steps: splitting recommendation service and experiment shunting to be independent; configuration management on recommended experiments in a web page form is supported, and experiment effect data are analyzed and displayed in real time; a MurmurHash algorithm which is high in calculation speed and small in conflict probability is adopted for shunting; a layering mechanism is adopted, so that the number of supported experiments is more; vertical flow is introduced, and a cross-multilayer experiment is supported; the shunting logic and the business logic are split and independent; the experiment shunting strategy supports a visual configuration pipe; a MurmurHash algorithm which is high in calculation speed and small in conflict probability is adopted; an overlapping, layering and barrel separating mechanism is adopted in the experiment, the number of experiments capable of being supported at the same time is remarkably increased, and the problem of insufficient flow is solved; unified experiment barrels are extracted from each layer to serve as a control group, and the actual effect comparison effect credibility is high. The Abtest shunting method of the overlapping, layering and barrel separating mechanism is characterized in that layering of experiments is supported, and corresponding experiment levels are selected according to different experiment types.

Description

Technical Field

[0001] The invention relates to the technical field of Abtest flow splitting methods, which is an Abtest flow splitting method with overlapping layered bucketing mechanism. Background Art

[0002] Abtest splitting is a method used in AB testing (also known as A / B testing) to distribute traffic (such as user visits and clicks) to different experimental groups and control groups.

[0003] The original diversion plan: the diversion logic is coupled with the business logic code; the traditional hashing method is used to divert users; all experiments are not stratified.

[0004] However, the existing traditional diversion technology has the following disadvantages and shortcomings:

[0005] 1. The diversion logic is highly coupled with the business logic code, resulting in poor system stability;

[0006] 2. Experimental strategy diversion does not support visual configuration and is completely black box;

[0007] 3. Traditional hashing methods are inefficient and can easily lead to traffic skew;

[0008] 4. The experiments are not layered, resulting in insufficient traffic, few experiments that can be supported at the same time, slow demand iteration, and low efficiency;

[0009] 5. There is no unified reference group for comparison, so the credibility of the actual results is low and it is not convincing. Summary of the invention

[0010] The present invention provides an Abtest diversion method with overlapping layered bucketing mechanism in order to solve the following technical problems: the diversion logic is highly coupled with the business logic code, resulting in poor system stability; the experimental strategy diversion does not support visual configuration and is completely black box; the traditional hash method is inefficient and easily leads to traffic tilt; the experiment is not layered, resulting in insufficient traffic, few experiments that can be supported simultaneously, slow demand iteration, and low efficiency; there is no unified reference group as a control, the credibility of the actual effect is low, and it is not persuasive.

[0011] The present invention solves the above technical problems through the following technical solutions:

[0012] The present invention provides an Abtest traffic diversion method with an overlapping layered bucketing mechanism, and the Abtest traffic diversion method with an overlapping layered bucketing mechanism comprises the following steps:

[0013] S1. Split the recommendation service and experiment flow into independent ones, so as to make the system architecture more reasonable and improve the system robustness;

[0014] S2. Supports configuration management of recommended experiments in the form of web pages, and real-time analysis and display of experimental effect data;

[0015] S3. Use the MurmurHash algorithm with fast calculation speed and low conflict probability to divert traffic, improve system performance, and avoid traffic tilt;

[0016] S4. The experiment adopts a layered mechanism, which supports more experiments and solves the problem of insufficient single-layer traffic;

[0017] S5. Introduce vertical traffic to support experiments across multiple layers;

[0018] S6. The diversion logic and business logic are separated and independent, the system structure is more reasonable and the system stability is strong;

[0019] S7. The experiment diversion strategy supports visual configuration management, which increases the participation of personnel in the recommended experiments, thereby greatly improving work efficiency;

[0020] S8. Use the MurmurHash algorithm with fast calculation speed and low conflict probability to improve system performance and avoid traffic skew;

[0021] S9. The experiment adopts an overlapping layered bucket mechanism, which significantly increases the number of experiments that can be supported simultaneously and solves the problem of insufficient traffic.

[0022] S10. A uniform experimental bucket is extracted from each layer as a control group, which makes the actual comparison effect more credible and enhances the scientific nature of the experiment.

[0023] Furthermore, in the stratified mechanism of step (S4), the traffic is divided into multiple layers according to different experimental requirements, which can be divided into a recommendation algorithm recall layer, a recommendation algorithm ranking layer, a scattered layer, or different page layers such as the home page, business page, and detail page, to ensure that the experiments in each layer are independent of each other, that is, the modified experimental parameters are not related to the observed product indicators.

[0024] Furthermore, the overlapping layered bucketing in step (S9) is to divide the traffic into multiple overlapping layers or buckets, and perform independent AB tests in each layer or bucket. Since many types of experiments are unrelated from the modified experimental parameters to the observed product indicators, the experiments can be divided into multiple independent layers. When conducting experiments in the same experimental layer, the traffic between each experiment does not overlap, which is also called mutual exclusion, and the traffic between layers is reused and overlapped.

[0025] Furthermore, in the step (S9), a certain traffic quota is allocated to each model (or strategy) according to the experimental requirements. The allocation of quota should be based on the experimental objectives and expected results to ensure that each model has sufficient traffic for testing.

[0026] Furthermore, in the step (S9), a corresponding experimental strategy is implemented in each bucket, and user behavior data is collected to ensure the accuracy and completeness of the data during the experiment.

[0027] Furthermore, in the step (S3), the MurmurHash algorithm converts the input data into hash values ​​through a series of bit operations and shift operations. These hash values ​​are evenly distributed in the hash space, so that different input data can be mapped to different hash values. In the diversion application, these hash values ​​can be used to distribute the input data to different processing paths or storage shards.

[0028] Furthermore, in the step (S4), the experiment adopts AB testing, and the MurmurHash algorithm can be used to hash and distribute users to different experimental groups. Since the MurmurHash algorithm has the characteristics of uniform distribution, low collision rate and high sensitivity, it can ensure the orthogonality between experimental groups and the uniformity of user distribution.

[0029] Furthermore, the web page in step (S2), i.e., a web page, is the basic element that constitutes a website. It is usually a file written in HTML (Hypertext Markup Language). These files contain text, images, links, scripts, and style sheet elements, which are used to display to users on a web browser. HTML is the core of a web page and is used to define the structure and content of the page.

[0030] Furthermore, in step (S4), the sample size required for each experimental group and control group is determined according to the experimental objectives and confidence level to ensure that the sample size is large enough to improve the accuracy and reliability of the experimental results.

[0031] Furthermore, the vertical traffic in the step (S5) generally refers to the user behavior or data flow in the vertical direction in a web page or application, and cross-layer experiments involve experimental designs conducted at different levels or modules. In web development, through reasonable page layout, attractive content presentation, and convenient interaction design, users can be guided to scroll down the page, increasing their stay time and number of interactions on the page.

[0032] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present invention.

[0033] The positive and progressive effects of the present invention are:

[0034] The Abtest diversion method with overlapping stratification and bucketing mechanism proposed above supports stratification of experiments and selects the corresponding experimental level according to different experimental types;

[0035] Supports overlapping layered bucketing, that is, the same level can dynamically expand the experimental level according to different experimental strategy types;

[0036] A bucketing strategy is adopted, that is, all traffic is evenly divided into 20 buckets, with 5% of the traffic in each bucket, to avoid experimental indicator errors caused by inconsistent traffic;

[0037] Introduce the concept of vertical flow, and observe the a-b effect by reversing the bucket. At the same time, vertical flow can be used to conduct experiments across multiple layers.

[0038] Through the visualization platform configuration management experiment and observation of experimental indicators, the efficiency of experimental iteration is greatly improved;

[0039] The statistical results of experimental indicators were verified by chi-square test to improve the confidence of experimental results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application.

[0041] Figure 1 The figure is a flow chart of the Abtest splitting method of the present invention.

[0042] Figure 2 This is a simplified diagram of the Abtest design concept of the Abtest diversion method of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0045] Embodiment 1:

[0046] like Figure 1-2 As shown, the Abtest traffic diversion method with overlapping layered bucketing mechanism includes the following steps:

[0047] S1. Split the recommendation service and experiment flow into independent ones, so as to make the system architecture more reasonable and improve the system robustness;

[0048] S2. Supports configuration management of recommended experiments in the form of web pages, and real-time analysis and display of experimental effect data;

[0049] S3. Use the MurmurHash algorithm with fast calculation speed and low conflict probability to divert traffic, improve system performance, and avoid traffic tilt;

[0050] S4. The experiment adopts a layered mechanism, which supports more experiments and solves the problem of insufficient single-layer traffic;

[0051] S5. Introduce vertical traffic to support experiments across multiple layers;

[0052] S6. The diversion logic and business logic are separated and independent, the system structure is more reasonable and the system stability is strong;

[0053] S7. The experiment diversion strategy supports visual configuration management, which increases the participation of personnel in the recommended experiments, thereby greatly improving work efficiency;

[0054] S8. Use the MurmurHash algorithm with fast calculation speed and low conflict probability to improve system performance and avoid traffic skew;

[0055] S9. The experiment adopts an overlapping layered bucket mechanism, which significantly increases the number of experiments that can be supported simultaneously and solves the problem of insufficient traffic.

[0056] S10. A uniform experimental bucket is extracted from each layer as a control group, which makes the actual comparison effect more credible and enhances the scientific nature of the experiment.

[0057] Furthermore, in the stratified mechanism of step (S4), the traffic is divided into multiple layers according to different experimental requirements, which can be divided into a recommendation algorithm recall layer, a recommendation algorithm ranking layer, a scattered layer, or different page layers such as the home page, business page, and detail page, to ensure that the experiments in each layer are independent of each other, that is, the modified experimental parameters are not related to the observed product indicators.

[0058] Furthermore, the overlapping layered bucketing in step (S9) is to divide the traffic into multiple overlapping layers or buckets, and perform independent AB tests in each layer or bucket. Since many types of experiments are unrelated from the modified experimental parameters to the observed product indicators, the experiments can be divided into multiple independent layers. When conducting experiments in the same experimental layer, the traffic between each experiment does not overlap, which is also called mutual exclusion, and the traffic between layers is reused and overlapped.

[0059] Furthermore, in the step (S9), a certain traffic quota is allocated to each model (or strategy) according to the experimental requirements. The allocation of quota should be based on the experimental objectives and expected results to ensure that each model has sufficient traffic for testing.

[0060] Furthermore, in the step (S9), a corresponding experimental strategy is implemented in each bucket, and user behavior data is collected to ensure the accuracy and completeness of the data during the experiment.

[0061] Furthermore, in the step (S3), the MurmurHash algorithm converts the input data into hash values ​​through a series of bit operations and shift operations. These hash values ​​are evenly distributed in the hash space, so that different input data can be mapped to different hash values. In the diversion application, these hash values ​​can be used to distribute the input data to different processing paths or storage shards.

[0062] Furthermore, in the step (S4), the experiment adopts AB testing, and the MurmurHash algorithm can be used to hash and distribute users to different experimental groups. Since the MurmurHash algorithm has the characteristics of uniform distribution, low collision rate and high sensitivity, it can ensure the orthogonality between experimental groups and the uniformity of user distribution.

[0063] Furthermore, the web page in step (S2), i.e., a web page, is the basic element that constitutes a website. It is usually a file written in HTML (Hypertext Markup Language). These files contain text, images, links, scripts, and style sheet elements, which are used to display to users on a web browser. HTML is the core of a web page and is used to define the structure and content of the page.

[0064] Furthermore, in step (S4), the sample size required for each experimental group and control group is determined according to the experimental objectives and confidence level to ensure that the sample size is large enough to improve the accuracy and reliability of the experimental results.

[0065] Furthermore, the vertical traffic in the step (S5) generally refers to the user behavior or data flow in the vertical direction in a web page or application, and cross-layer experiments involve experimental designs conducted at different levels or modules. In web development, through reasonable page layout, attractive content presentation, and convenient interaction design, users can be guided to scroll down the page, increasing their stay time and number of interactions on the page.

[0066] The Abtest traffic diversion method with the above-mentioned overlapping layered bucketing mechanism supports the stratification of experiments and selects the corresponding experimental level according to the different experimental types; supports overlapping layered bucketing, that is, the same level can dynamically expand the experimental level according to the different experimental strategy types; adopts a bucketing strategy, that is, all traffic is evenly divided into 20 buckets, each bucket has 5% traffic, to avoid experimental indicator errors caused by inconsistent traffic; introduces the concept of vertical traffic, and looks at the ab effect by reversing the bucket, and uses vertical traffic to conduct experiments across multiple layers.

[0067] Embodiment 1:

[0068] like Figure 1-2 As shown, the Abtest traffic diversion method with overlapping layered bucketing mechanism includes the following steps:

[0069] S1. Split the recommendation service and experiment flow into independent ones, so as to make the system architecture more reasonable and improve the system robustness;

[0070] S2. Supports configuration management of recommended experiments in the form of web pages, and real-time analysis and display of experimental effect data;

[0071] S3. Use the MurmurHash algorithm with fast calculation speed and low conflict probability to divert traffic, improve system performance, and avoid traffic tilt;

[0072] S4. The experiment adopts a layered mechanism, which supports more experiments and solves the problem of insufficient single-layer traffic;

[0073] S5. Introduce vertical traffic to support experiments across multiple layers;

[0074] S6. The diversion logic and business logic are separated and independent, the system structure is more reasonable and the system stability is strong;

[0075] S7. The experiment diversion strategy supports visual configuration management, which increases the participation of personnel in the recommended experiments, thereby greatly improving work efficiency;

[0076] S8. Use the MurmurHash algorithm with fast calculation speed and low conflict probability to improve system performance and avoid traffic skew;

[0077] S9. The experiment adopts an overlapping layered bucket mechanism, which significantly increases the number of experiments that can be supported simultaneously and solves the problem of insufficient traffic.

[0078] S10. A uniform experimental bucket is extracted from each layer as a control group, which makes the actual comparison effect more credible and enhances the scientific nature of the experiment.

[0079] Furthermore, in the stratified mechanism of step (S4), the traffic is divided into multiple layers according to different experimental requirements, which can be divided into a recommendation algorithm recall layer, a recommendation algorithm ranking layer, a scattered layer, or different page layers such as the home page, business page, and detail page, to ensure that the experiments in each layer are independent of each other, that is, the modified experimental parameters are not related to the observed product indicators.

[0080] Furthermore, the overlapping layered bucketing in step (S9) is to divide the traffic into multiple overlapping layers or buckets, and perform independent AB tests in each layer or bucket. Since many types of experiments are unrelated from the modified experimental parameters to the observed product indicators, the experiments can be divided into multiple independent layers. When conducting experiments in the same experimental layer, the traffic between each experiment does not overlap, which is also called mutual exclusion, and the traffic between layers is reused and overlapped.

[0081] Furthermore, in the step (S9), a certain traffic quota is allocated to each model (or strategy) according to the experimental requirements. The allocation of quota should be based on the experimental objectives and expected results to ensure that each model has sufficient traffic for testing.

[0082] Furthermore, in the step (S9), a corresponding experimental strategy is implemented in each bucket, and user behavior data is collected to ensure the accuracy and completeness of the data during the experiment.

[0083] Furthermore, in the step (S3), the MurmurHash algorithm converts the input data into hash values ​​through a series of bit operations and shift operations. These hash values ​​are evenly distributed in the hash space, so that different input data can be mapped to different hash values. In the diversion application, these hash values ​​can be used to distribute the input data to different processing paths or storage shards.

[0084] Furthermore, in the step (S4), the experiment adopts AB testing, and the MurmurHash algorithm can be used to hash and distribute users to different experimental groups. Since the MurmurHash algorithm has the characteristics of uniform distribution, low collision rate and high sensitivity, it can ensure the orthogonality between experimental groups and the uniformity of user distribution.

[0085] Furthermore, the web page in step (S2), i.e., a web page, is the basic element that constitutes a website. It is usually a file written in HTML (Hypertext Markup Language). These files contain text, images, links, scripts, and style sheet elements, which are used to display to users on a web browser. HTML is the core of a web page and is used to define the structure and content of the page.

[0086] Furthermore, in step (S4), the sample size required for each experimental group and control group is determined according to the experimental objectives and confidence level to ensure that the sample size is large enough to improve the accuracy and reliability of the experimental results.

[0087] Furthermore, the vertical traffic in the step (S5) generally refers to the user behavior or data flow in the vertical direction in a web page or application, and cross-layer experiments involve experimental designs conducted at different levels or modules. In web development, through reasonable page layout, attractive content presentation, and convenient interaction design, users can be guided to scroll down the page, increasing their stay time and number of interactions on the page.

[0088] The Abtest diversion method of the above-mentioned overlapping stratified bucketing mechanism supports stratification of experiments and selects the corresponding experimental level according to different experimental types; supports overlapping stratified bucketing, that is, the same level can dynamically expand the experimental level according to different experimental strategy types; adopts a bucketing strategy, that is, all traffic is evenly divided into 20 buckets, each bucket has 5% traffic, to avoid experimental indicator errors caused by inconsistent traffic; introduces the concept of vertical traffic, and looks at the ab effect by reversing the bucket, and at the same time uses vertical traffic to do experiments across multiple layers; configures and manages experiments through a visualization platform and observes experimental indicators, greatly improving the efficiency of experimental iterations; verifies the statistical results of experimental indicators through the chi-square test to improve the confidence of experimental effects.

[0089] The circuits, electronic components and modules involved are all prior art and can be fully implemented by those skilled in the art. Needless to say, the content protected by this application does not involve improvements to software and methods.

[0090] The present invention is not limited to the above-mentioned embodiments. Any changes in shape or structure are within the protection scope of the present invention. The protection scope of the present invention is defined by the attached claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principle and essence of the present invention, but these changes and modifications are within the protection scope of the present invention.

Claims

1. An Abtest traffic diversion method with overlapping layered bucketing mechanism, characterized in that: The Abtest traffic diversion method with overlapping layered bucketing mechanism comprises the following steps: S1. Split the recommendation service and experiment flow into independent ones, so as to make the system architecture more reasonable and improve the system robustness; S2. Supports configuration management of recommended experiments in the form of web pages, and real-time analysis and display of experimental effect data; S3. Use the MurmurHash algorithm with fast calculation speed and low conflict probability to divert traffic, improve system performance, and avoid traffic tilt; S4. The experiment adopts a layered mechanism, which supports more experiments and solves the problem of insufficient single-layer traffic; S5. Introduce vertical traffic to support experiments across multiple layers; S6. The diversion logic and business logic are separated and independent, the system structure is more reasonable and the system stability is strong; S7. The experiment diversion strategy supports visual configuration management, which increases the participation of personnel in the recommended experiments, thereby greatly improving work efficiency; S8. Use the MurmurHash algorithm with fast calculation speed and low conflict probability to improve system performance and avoid traffic skew; S9. The experiment adopts an overlapping layered bucket mechanism, which significantly increases the number of experiments that can be supported simultaneously and solves the problem of insufficient traffic. S10. A uniform experimental bucket is extracted from each layer as a control group, which makes the actual comparison effect more credible and enhances the scientific nature of the experiment.

2. The Abtest traffic diversion method with overlapping layered bucketing mechanism as claimed in claim 1, characterized in that: In the step (S4), the stratified mechanism divides the traffic into multiple layers according to different experimental requirements, which can be divided into a recommendation algorithm recall layer, a recommendation algorithm ranking layer, a scattered layer, or different page layers such as the home page, the business page, and the detail page, to ensure that the experiments in each layer are independent of each other, that is, the modified experimental parameters are not related to the observed product indicators.

3. The Abtest traffic diversion method with overlapping layered bucketing mechanism as claimed in claim 1, characterized in that: In the step (S9), the overlapping layered bucketing is to divide the traffic into multiple overlapping layers or buckets, and perform independent AB tests in each layer or bucket. Since many types of experiments are unrelated from the modified experimental parameters to the observed product indicators, the experiments can be divided into multiple independent layers. When conducting experiments in the same experimental layer, the traffic between each experiment does not overlap, which is also called mutual exclusion, and the traffic between layers is reused and overlapped.

4. The Abtest traffic diversion method with overlapping layered bucketing mechanism as claimed in claim 1, characterized in that: In the step (S9), a certain traffic quota is allocated to each model (or strategy) according to the experimental requirements. The allocation of quota should be based on the experimental objectives and expected results to ensure that each model has sufficient traffic for testing.

5. The Abtest traffic diversion method with overlapping layered bucketing mechanism as claimed in claim 1, characterized in that: In the step (S9), a corresponding experimental strategy is implemented in each bucket, and user behavior data is collected to ensure the accuracy and completeness of the data during the experiment.

6. The Abtest traffic diversion method with overlapping layered bucketing mechanism as claimed in claim 1, characterized in that: In the step (S3), the MurmurHash algorithm converts the input data into hash values ​​through a series of bit operations and shift operations. These hash values ​​are evenly distributed in the hash space, so that different input data can be mapped to different hash values. In the diversion application, these hash values ​​can be used to distribute the input data to different processing paths or storage shards.

7. The Abtest traffic diversion method with overlapping layered bucketing mechanism as claimed in claim 1, characterized in that: In the step (S4), the experiment adopts AB test, and the MurmurHash algorithm can be used to hash and distribute users to different experimental groups. Since the MurmurHash algorithm has the characteristics of uniform distribution, low collision rate and high sensitivity, it can ensure the orthogonality between experimental groups and the uniformity of user distribution.

8. The Abtest traffic diversion method with overlapping layered bucketing mechanism as claimed in claim 1, characterized in that: The web page in step (S2), i.e., a web page, is a basic element constituting a website, and is usually a file written in HTML (Hypertext Markup Language). These files contain text, images, links, scripts, and style sheet elements, and are used to display to users on a web browser. HTML is the core of a web page and is used to define the structure and content of the page.

9. The Abtest traffic diversion method with overlapping layered bucketing mechanism as claimed in claim 1, characterized in that: In the step (S4), the sample size required for each experimental group and control group is determined according to the experimental objectives and confidence level to ensure that the sample size is large enough to improve the accuracy and reliability of the experimental results.

10. The Abtest traffic diversion method with overlapping layered bucketing mechanism as claimed in claim 1, characterized in that: The vertical traffic in the step (S5) generally refers to the user behavior or data flow in the vertical direction in a Web page or application, and the cross-layer experiment involves the experimental design conducted at different levels or modules. In Web development, through reasonable page layout, attractive content presentation and convenient interaction design, users can be guided to scroll down the page, thereby increasing their stay time and number of interactions on the page.

Images