Method and system for establishing AB experiment evaluation system
By creating characteristic layers and experimental layers in the AB experimental evaluation system and performing orthogonal shunts, the problem of excessive group differences caused by unbalanced flow distribution in traditional A/B experiments is solved, and the accuracy of the AB experimental results is achieved.
Patent Information
- Application Number
- CN202410562808.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-08
- Publication Date
- 2025-03-25
AI Technical Summary
The traditional A/B experiments' diverting method cannot ensure that there is no difference in flow rates between the experimental group and the control group, resulting in too large group differences caused by unbalanced flow distribution, which can easily lead to misjudgment of experimental results.
By creating characteristic layers and experimental layers in the AB experimental evaluation system, the user data is divided into orthogonal diversion according to the experimental layer, the control group and experimental group of the AB experiment are determined based on the characteristics, and the results of the AB experiment are recorded, and the significance of the AB experiment is finally tested.
It has achieved the establishment of indiscriminate experimental groups and control groups in different business scenarios, avoiding misjudgment caused by unbalanced traffic allocation, and ensuring the accuracy of AB experimental results.
Smart Images
Figure CN119693036A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer technology, and specifically relates to a method and system for establishing an AB experiment evaluation system. Background Art
[0002] At present, A / B experiments have become the "gold standard" for many Internet companies to evaluate their new product strategies and methods. In C-end business scenarios, it is widely used in business scenarios such as growth strategy, pricing strategy, experience optimization, performance optimization estimation, etc., to make data-driven decisions for strategy iteration. In particular, strategies in multiple scenarios such as regions, time slices, and users influence each other, and requests are not independent of each other in A / B experiments. A / B experiments are a "weapon" for refined iteration. By formulating two or more versions of solutions for the same iteration goal, in the same time dimension, A / B groups with the same or similar components adopt these versions respectively, and then collect experience data and business data from each group, and finally analyze and evaluate the best version to help the business make correct decisions and make iterations evolve in a better direction. Based on this, building an A / B test evaluation system suitable for C-end business came into being.
[0003] The traditional A / B experiment traffic diversion method cannot ensure that the traffic of the two groups, the experimental group and the control group, is the same. It cannot avoid the problem of excessive differences between the A / B groups due to unbalanced traffic distribution, which can easily lead to misjudgment of the experimental results. Summary of the invention
[0004] In order to solve the above problems, the present invention adopts the following technical solutions:
[0005] A method for establishing an AB experiment evaluation system, comprising the steps of:
[0006] Create features based on the target scenario, establish feature layers, and assign features to the feature layers; the features are events that are assumed to be related to user click behaviors;
[0007] Determine relevant experimental events of the target scenario, and assign the relevant experimental events to different experimental layers; the relevant experimental events are events related to user click behaviors;
[0008] Divide the user data into orthogonal diversions according to the experimental layer and then enter the feature layer. Determine the control group and experimental group of the AB experiment based on the characteristics, and record the results of the AB experiment.
[0009] According to the results of the AB experiment, the significance of the AB experiment was tested.
[0010] As a further solution of the present invention, the characteristics include red envelope lottery event, and the red envelope lottery event has at least two categories.
[0011] As a further solution of the present invention, the user data is orthogonally split into the experimental layer and then enters the characteristic layer, the control group and the experimental group of the AB experiment are determined according to the characteristics, and the AB experiment results are recorded, including:
[0012] Ensure that the source channel, promotion bid, and user attributes of the user data are consistent through the controller;
[0013] Orthogonally split the user data into the experimental layer and then into the feature layer;
[0014] Determine the control group and experimental group of the AB experiment based on the red envelope lottery event, and record the results of the AB experiment.
[0015] As a further solution of the present invention, the AB experiment results include: the number of clicks pv, the number of clicks uv and the click rate of the red envelope lottery event.
[0016] As a further solution of the present invention, the testing of the significance of the AB experiment according to the result of the AB experiment comprises:
[0017] Calculate the activity participation rate, prize collection rate, recharge / payment click rate, recharge / payment order conversion rate and recharge / payment total amount of the red envelope lottery event based on the click volume pv, click volume uv and click rate of the red envelope lottery event;
[0018] The significance of the AB experiment is tested based on the activity participation rate, prize collection rate, recharge payment click rate, recharge order conversion rate and total recharge amount of the red envelope lottery event.
[0019] As a further solution of the present invention, it is characterized in that the relevant experimental events include: urban area, GeoHash, time slice and user data.
[0020] As a further solution of the present invention, the step of orthogonally splitting the user data into the experimental layer and then entering the characteristic layer includes: processing the user data according to the MurmurHash3 hash algorithm.
[0021] A system for implementing the above method includes the following modules:
[0022] Hierarchical module: used to create features based on target scenarios, establish feature layers, and assign features to feature layers; the features are events that are assumed to be related to user click behaviors;
[0023] Used to determine relevant experimental events of the target scenario, and allocate the relevant experimental events to different experimental layers; the relevant experimental events are events related to user click behaviors;
[0024] Diversion calculation module: used to divide user data into orthogonal diversions according to the experimental layer and then enter the feature layer, determine the control group and experimental group of the AB experiment according to the characteristics, and record the results of the AB experiment;
[0025] Evaluation module: according to the results of the AB experiment, the significance of the AB experiment is tested.
[0026] A server comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the method is implemented when the processor executes the program.
[0027] A computer-readable storage medium stores a computer program, which implements the above method when executed by a processor.
[0028] Beneficial effects of the present invention:
[0029] A method for establishing an AB experiment evaluation system and its system are intended to address the current status of technical defects, meet the demands of different business scenarios, produce an A / B platform with multiple diversion strategies, and conduct A / B experiments in scenarios where strategies influence each other and requests are not independent. A method for establishing an AB experiment evaluation system adopts a diversion method with limited access. Different diversion factors are selected for different experiments, that is, user data is divided into feature layers and multiple experimental layers. Samples are taken independently and randomly from different experimental layers, mixed with the feature layers, and randomly sampled again to establish undifferentiated experimental groups and control groups as the basis for AB experiment diversion configuration. When the results of the AB experiment are not statistically significant, it can be considered that the separated experimental and control groups are indistinguishable. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is a flow chart of a method for establishing an AB experimental evaluation system;
[0031] Figure 2 Another schematic diagram of a method for establishing an AB experiment evaluation system;
[0032] Figure 3 A schematic diagram of an AB experiment flow distribution model for a method of establishing an AB experiment evaluation system;
[0033] Figure 4 A schematic diagram of another AB experiment flow distribution model for a method of establishing an AB experiment evaluation system;
[0034] Figure 5 A schematic diagram of the structure of a system for establishing an AB experiment evaluation system. DETAILED DESCRIPTION
[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. It should be understood that the present application is not limited to the example embodiments disclosed herein. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0036] Example 1
[0037] like Figure 1 As shown, a method for establishing an AB experiment evaluation system comprises the following steps:
[0038] S1: Create features based on the target scenario, establish feature layers, and assign features to feature layers; features are events that are assumed to be related to user click behaviors;
[0039] S2: Determine the relevant experimental events of the target scenario and assign the relevant experimental events to different experimental layers; the relevant experimental events are events related to the user's click behavior;
[0040] S3: Divide the user data into orthogonal diversions according to the experimental layer and then enter the feature layer. Determine the control group and experimental group of the AB experiment according to the characteristics and record the results of the AB experiment.
[0041] S4: Based on the results of AB experiment, test the significance of AB experiment.
[0042] Specifically, features refer to experimental features developed for various specific functions or activities such as playback, activities, recharge, landing page styles, etc. The most strategic variants are selected by verifying the differences in algorithm recommendations, system display interfaces, and user interactions through feature variables.
[0043] In this embodiment, user data is orthogonally split at a conventional experimental layer and then mixed with the characteristic layer for calculation to test the significance of the AB experiment.
[0044] In this embodiment, the user data is processed according to the MurmurHash3 hash algorithm.
[0045] A method for establishing an AB experiment evaluation system, which involves independently and randomly sampling from different experimental layers, mixing with the characteristic layer and randomly sampling again, to establish indifferent experimental and control groups as the basis for AB experiment diversion configuration. When the results of the AB experiment are not statistically significant, it can be considered that the separated experimental and control groups are indifferent.
[0046] In a specific embodiment, for example, the target scenario is a promotional activity, the feature is a red envelope lottery event, and the red envelope lottery event has at least two categories, including "get 10 yuan for every 199 yuan spent" and "get 30 yuan for every 399 yuan spent". The experimental layers include the city area layer, GeoHash layer, time slice layer, user layer, and feature layer.
[0047] like Figure 3 The traffic distribution model diagram shown in the figure, diversion: refers to directly cutting the overall user data into several pieces. User data can only be in one experiment and will not affect each other. Even distribution of traffic on the same layer: users in the same experimental layer participate in the same experiment, and the experiments are mutually exclusive. Layering: refers to placing the same batch of user data in different buckets after continuous randomization. The same user data belongs to multiple different experiments and does not affect each other. Orthogonal distribution of traffic on different layers: user data layering orthogonality can support multiple experiments at the same time, such as sorting by different content at the same time. The meaning of orthogonality is that each experiment is independent of each other, that is, users are re-scattered at each layer.
[0048] like Figure 2 The method for establishing an AB experimental evaluation system shown includes the following steps:
[0049] S1: Create a red envelope lottery event based on the promotional activity and establish a red envelope lottery event feature layer.
[0050] Specifically, the features "Get 10 yuan for purchases over 199 yuan" and "Get 30 yuan for purchases over 399 yuan" are divided into feature layers.
[0051] S2: Determine the relevant experimental events of the promotion and assign the relevant experimental events to different experimental layers.
[0052] Specifically, the relevant experimental events include: city area, GeoHash, time slice and user. The city area, time slice and user data are independently allocated to each experimental layer, that is, the experimental layer includes the city area layer, GeoHash layer, time slice layer and user layer.
[0053] Among them, the city area layer means dividing the city into regions; the GeoHash layer means encoding a region into a one-dimensional string using two-dimensional longitude and latitude; the time slice layer means dividing time into time periods for implementing experimental strategy scheduling; and the user layer means dividing users into hash buckets based on the userId and device ID.
[0054] A traffic admission diversion method is adopted. Each experimental layer calculates the hit traffic slot according to the diversion configuration information and user data in the layer, and then circles the experiment of the traffic slot based on the traffic slot hit. The experiment determines the experimental traffic ratio according to the number of traffic slots it has.
[0055] S3: Divide the user data into orthogonal diversions according to the experimental layer and then enter the feature layer. Determine the control group and experimental group of the AB experiment according to the characteristics and record the results of the AB experiment; including:
[0056] S301: Ensure that the source channel of user data, promotion bids and user attributes are consistent through the controller.
[0057] Specifically, add a controller as an experimental layer, use the controller to combine the IDs of the three-party ad slots indiscriminately, configure different loading strategies, floor price strategies, etc., and verify the ad revenue. Through the controller layer, select effective controller strategy variants to avoid deviations in the user source, which may cause errors in the experimental results.
[0058] like Figure 4 As shown, the controller of each group of experiments ensures that the source channel, promotion bid and user attributes of user data are consistent. That is, each independent AB experiment must ensure that the source traffic attributes split by the controller are the same, and ensure that in the AB experiment, the fluctuations are caused by the diversion group, not the strategy. For example, controller X is to screen the female group, controller Y is to screen the male group, and controller Z is to screen the elderly group. Then all of them will enter the experiment as a red envelope lottery event to verify the impact of the feature experiments of "get 10 yuan for purchases over 199 yuan" and "get 30 yuan for purchases over 399 yuan" on the sales of cosmetics. The experimental results obtained are caused by different users, not by different strategies.
[0059] S302: dividing the user data into orthogonal streams according to the experimental layer and then entering the feature layer;
[0060] Specifically, user data is extracted independently and randomly from different experimental layers, mixed with the feature layer and randomly extracted again, and undifferentiated experimental and control groups are established as the basis for AB experimental diversion configuration. Experiments in the same layer are mutually exclusive and will not interfere with each other.
[0061] S303: Determine the control group and experimental group of the AB experiment according to the red envelope lottery event, and record the results of the AB experiment.
[0062] The AB experiment results include: the number of clicks pv, number of clicks uv and click-through rate of the red envelope lottery event.
[0063] Among them, "PV" in traffic statistics stands for visit volume. Specifically, it refers to the number of page clicks or views of the website, or the number of page refreshes. Every time a website page is refreshed, a "PV" is counted. If a user refreshes the page 100 times, then the traffic statistics tool will show 100 "PV" volumes.
[0064] "UV" in traffic statistics refers to a client that visits a website, including PC and mobile clients. Within 24 hours of the same day, the same "UV" client that visits the website can only be counted once no matter how many times it visits.
[0065] S4: Based on the results of the AB experiment, test the significance of the AB experiment, including:
[0066] S401: Calculate the activity participation rate, prize collection rate, recharge / payment click rate, recharge / order conversion rate and recharge / order total amount of the red envelope lottery event based on the click volume pv, click volume uv and click rate of the red envelope lottery event.
[0067] S402: Test the significance of the AB experiment based on the activity participation rate, prize collection rate, recharge payment click rate, recharge order conversion rate and total recharge amount of the red envelope lottery event.
[0068] Because the red envelope prizes for the lottery have different discounts, after the user clicks to receive the prize, the experiment is used to verify the impact of the discount on the activity participation rate, prize collection rate, recharge / payment click rate, recharge / payment order conversion rate, and the total recharge / payment amount.
[0069] In a specific embodiment, before the AB experiment, AA grouping is performed to find out the indifferent experimental group and control group, which serves as the basis for our AB experiment diversion configuration.
[0070] Example 2
[0071] Based on the same inventive concept as the method for establishing an AB experiment evaluation system in the aforementioned embodiment, the embodiment of this specification also provides a system, such as Figure 5 As shown, it includes the following modules:
[0072] Hierarchical module: used to create features based on target scenarios, establish feature layers, and assign features to feature layers; the features are events that are assumed to be related to user click behaviors;
[0073] Used to determine relevant experimental events of the target scenario, and allocate the relevant experimental events to different experimental layers; the relevant experimental events are events related to user click behaviors;
[0074] Diversion calculation module: used to divide user data into orthogonal diversions according to the experimental layer and then enter the feature layer, determine the control group and experimental group of the AB experiment according to the characteristics, and record the results of the AB experiment;
[0075] Evaluation module: according to the results of the AB experiment, the significance of the AB experiment is tested.
[0076] In this embodiment, the system divides the target scenario into characteristic layers and multiple experimental layers in the stratification module, and then divides the calculation module to implement the AB experiment. Finally, the AB experiment results enter the evaluation module to test the significance of the AB experiment. Among them, in the calculation module, the AA experiment can be carried out first, and then the AB experiment. The control group and experimental group configuration of the AA experiment and the AB experiment are the same, and the random volatility of the test indicator itself is tested.
[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above and the corresponding beneficial effects can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0078] Example 3
[0079] Based on the same inventive concept as the method for establishing an AB experiment evaluation system in the aforementioned embodiment, the embodiment of this specification also provides a server, including a memory processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the aforementioned AB experiment data processing method when executing the program.
[0080] Example 4
[0081] Based on the inventive concept of the method for establishing an AB experimental evaluation system in the aforementioned embodiment, the embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by the processor to implement the steps in the method for establishing an AB experimental evaluation system in the preceding text. In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, a process or function according to an embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, computer instructions can be transmitted from a website site, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).
[0082] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0083] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for establishing an AB experiment evaluation system, characterized in that the steps include: Create features based on the target scenario, establish feature layers, and assign features to the feature layers; the features are events that are assumed to be related to user click behaviors; Determine relevant experimental events of the target scenario, and assign the relevant experimental events to different experimental layers; the relevant experimental events are events related to user click behaviors; Divide the user data into orthogonal diversions according to the experimental layer and then enter the feature layer. Determine the control group and experimental group of the AB experiment based on the characteristics, and record the results of the AB experiment. According to the results of the AB experiment, the significance of the AB experiment was tested.
2. The method for establishing an AB experiment evaluation system according to claim 1, characterized in that: The characteristics include a red envelope lottery event, and the red envelope lottery event has at least two categories.
3. The method for establishing an AB experiment evaluation system according to claim 2, characterized in that: The user data is orthogonally split into the experimental layer and then enters the feature layer, the control group and the experimental group of the AB experiment are determined according to the characteristics, and the AB experiment results are recorded, including: Ensure that the source channel, promotion bid, and user attributes of the user data are consistent through the controller; Orthogonally split the user data into the experimental layer and then into the feature layer; Determine the control group and experimental group of the AB experiment based on the red envelope lottery event, and record the results of the AB experiment.
4. The method for establishing an AB experiment evaluation system according to claim 3, characterized in that: The AB experiment results include: the number of clicks pv, the number of clicks uv and the click rate of the red envelope lottery event.
5. The method for establishing an AB experiment evaluation system according to claim 4, characterized in that: According to the result of the AB experiment, testing the significance of the AB experiment includes: Calculate the activity participation rate, prize collection rate, recharge / payment click rate, recharge / payment order conversion rate and recharge / payment total amount of the red envelope lottery event based on the click volume pv, click volume uv and click rate of the red envelope lottery event; The significance of the AB experiment is tested based on the activity participation rate, prize collection rate, recharge payment click rate, recharge order conversion rate and total recharge amount of the red envelope lottery event.
6. The method for establishing an AB experiment evaluation system according to any one of claims 1 to 5, characterized in that: The relevant experimental events include: urban area, GeoHash, time slice and user data.
7. The method for establishing an AB experiment evaluation system according to claim 1, characterized in that: The step of orthogonally splitting the user data into the experimental layer and then entering the characteristic layer includes: processing the user data according to the MurmurHash3 hash algorithm.
8. A system for implementing the method according to any one of claims 1 to 7, characterized in that: Includes the following modules: Hierarchical module: used to create features based on target scenarios, establish feature layers, and assign features to feature layers; the features are events that are assumed to be related to user click behaviors; Used to determine relevant experimental events of the target scenario, and allocate the relevant experimental events to different experimental layers; the relevant experimental events are events related to user click behaviors; Diversion calculation module: used to divide user data into orthogonal diversions according to the experimental layer and then enter the feature layer, determine the control group and experimental group of the AB experiment according to the characteristics, and record the results of the AB experiment; Evaluation module: according to the results of the AB experiment, the significance of the AB experiment is tested.
9. A server, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the method according to any one of claims 1 to 7 is implemented when the processor executes the program.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.