Fast grouping method and system applied to AB test based on flow

Abstract

The invention discloses a rapid grouping method and system applied to an AB test based on flow, and belongs to the technical field of data processing, and the method comprises the steps: selecting samples, carrying out covariant extension, and then carrying out random sorting, so as to form a sequence composed of a plurality of sample batches with the same size; respectively distributing the sample batches with the same size into the experimental group and the control group, and obtaining the sample batch corresponding to the current distribution progress; distributing a first subset of the sample batch into an experimental group or a control group; distributing the second subset to a group different from the first subset; and according to the mahalanobis distance between the two distribution schemes of each distributed batch, determining a specific scheme that the sample batch is distributed to the experimental group and the control group. The method supports grouping balance division under the constraint of any multiple covariables.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a fast grouping method and system for a flow-based AB test. Background Art

[0002] In Internet business scenarios, traffic-based AB experiments usually use static lists to divide groups. However, in actual experiments, it is usually unpredictable which samples in the list will be activated due to the nature of the traffic itself. The groups we need to divide should at least be balanced in the performance of the core statistical indicators of concern, and at least there will not be drastic fluctuations during the experiment period. Otherwise, the reliability and validity of the AB experiment cannot be guaranteed.

[0003] The simplest experimental grouping method is completely random division. This method randomly places samples from the population into a given group, ensuring only that there is no non-empty intersection of samples between groups. This grouping method is very likely to form a de facto unbalanced sub-sampling group due to the activation and inactivation of samples, that is, the results of core statistical indicators during the experimental observation period will show significant differences in each group.

[0004] The improved method is regrouping. This method calculates the difference between groups of the statistical indicators of interest under each division scheme based on completely random division. If the difference is greater than the threshold, the grouping is re-performed until the difference between groups of the indicators under a grouping scheme is less than the preset value or reaches the preset retry limit. This method is relatively simple and there is a chance to encounter a suitable grouping, but it takes a long time to process tens of thousands or even larger sample groups.

[0005] An improvement based on regrouping, the mbar method, i.e., small batch adaptive grouping, first removes abnormal values ​​such as outliers from the sample population, then divides the sample population into blocks in a cross-dimensional manner according to the statistical indicators of interest, and further divides the sample ID in each block into batch × grouping blocks by hashing, usually two groups, and then uses the idea of ​​regrouping to randomly select a permutation scheme for the pre-divided sub-blocks in each batch and assign it to the existing grouping scheme, and determines whether it is appropriate according to the regrouping method, and completes the allocation of all batch data blocks in turn. This method is usually suitable for the division of samples above one million. When the sample size is low, there is no advantage in time complexity, and the number of statistical indicators of interest cannot be large, usually only one, which is acceptable for massive samples, but when the sample size is lower than this level, the possibility of imbalance in the covariates representing various influencing factors in the grouping increases due to the reduction in samples, so the balance of key indicators in the future cannot be guaranteed based on historical statistics.

[0006] In order to support multiple covariates to achieve inter-group balance at the same time, the prior art proposes the PSR method, i.e. paired sequential random grouping. This method first randomly arranges all samples into a sequence, then takes two samples from the sequence each time and puts them into a given group, then calculates the Mahalanobis distance of the two groups based on the vector composed of all covariates, then exchanges samples, and calculates the Mahalanobis distance of the two groups again. If the second Mahalanobis distance is less than the first, the second division scheme is used; if it is greater, the first is used; if they are equal, a random assignment is performed. This method can achieve balanced group division under the constraints of almost any number of covariates, that is, the group balance of each covariate dimension can be guaranteed. However, the time complexity of this method is acceptable for samples below 10,000, but the grouping task of 100,000 samples takes several hours.

[0007] In summary, the existing technologies have the following shortcomings: the random method has a high probability of unbalanced grouping, which introduces noise into the experiment; the random regrouping method (CR) requires repeated attempts, has high time complexity, and the results are still not guaranteed to be appropriate; the small batch adaptive grouping method (MBAR) is designed for samples above one million, only considers the balance of one core variable in the grouping, and is not suitable for the simultaneous balanced division of multiple covariates. The paired sequential random grouping method (PSR) takes a long time to process the division of samples at the scale of 100,000, and the cost of grouping is too high. Therefore, it is urgent to design an improved method to achieve balanced grouping of 100,000 samples, support any number of covariate balance constraints, and have acceptable time complexity. Summary of the invention

[0008] In order to solve the above problems, the purpose of the present invention is to provide a rapid grouping technology applied to flow-based AB tests, so as to solve the technical problems that the existing technical methods either cannot achieve the balance of grouping on multiple covariates, or have high computational complexity and take up to several hours when processing the grouping of hundreds of thousands to millions of samples.

[0009] In order to achieve the above technical objectives, the present application provides a rapid grouping method for a flow-based AB test, comprising the following steps: After selecting samples for covariate expansion, they are randomly sorted to form a sequence consisting of several sample batches of the same size; Each sample batch of the same size is divided into two subsets of the same size, and the subsets are respectively assigned to the experimental group and the control group, and the sample batch corresponding to the current allocation progress is obtained; Assign the first subset of the sample batch to the experimental group or the control group; Assign the second subset of the sample batch to a group different from the first subset; According to the Mahalanobis distance between the two allocation schemes of each allocated batch, the specific scheme for allocating the sample batch to the experimental group and the control group is determined to ensure that the data of the experimental group and the control group are balanced in the covariate dimension.

[0010] Preferably, in the process of expanding the sample covariates, the numerical covariates are kept as they are, and the categorical variables are encoded in a one-hot manner, and expanded to covariates, where n is the number of categorical variables, is the number of categories in each categorical variable i.

[0011] Preferably, in the process of controlling the sample batches to be allocated to the experimental group and the control group for balanced allocation, a first subset in the sample batch is allocated to the experimental group, and a second subset in the sample batch is allocated to the control group, and a first Mahalanobis distance between the experimental group and the control group after the allocation is updated is obtained; Assign the first subset of the sample batch to the control group, and assign the second subset of the sample batch to the experimental group, and obtain the updated second Mahalanobis distance between the experimental group and the control group; According to the relationship between the first Mahalanobis distance and the second Mahalanobis distance, two subsets in the control sample batch are allocated to the specific scheme of balanced allocation between the experimental group and the control group.

[0012] Preferably, the allocation scheme for determining the two subsets of the sample batch and the corresponding relationship between the relative sizes of the first Mahalanobis distance and the second Mahalanobis distance are specifically expressed as follows: if ,but ; if ,but ; if ,but ; Where 0.75<q<1, distribution To ensure that the two subsets are not assigned to the experimental group or the control group at the same time, i represents the sample of the i-th batch, 2i+1 and 2i+2 represent the two subsets of the batch, M1 represents the first Mahalanobis distance, M2 represents the second Mahalanobis distance, and P() represents the probability of assigning a given subset to the experimental group.

[0013] Preferably, in the process of controlling sample batches to be allocated to the experimental group and the control group in equal proportion, when the number of sample batches is an odd number, the last sample batch is controlled to be allocated to the two groups with equal probability.

[0014] The present invention discloses a rapid grouping system for AB test based on flow, comprising: The data processing module is used to select samples for covariate expansion and then randomly sort them to form a sequence consisting of several sample batches of the same size; The data allocation module is used to allocate sample batches of the same size to the experimental group and the control group respectively, and obtain the sample batches corresponding to the current allocation progress; A first data grouping module is used to allocate the first subset of the sample batch to the experimental group and the second subset to the control group; A second data grouping module is used to allocate the first subset of the sample batch to the control group and the second subset to the experimental group; The allocation control module is used to control the allocation of the sample batches to the experimental group and the control group in a balanced manner according to the Mahalanobis distance between the first scheme of the allocated sample batches and the second scheme of the sample batches.

[0015] Preferably, the data processing module is also used to keep the numerical covariates intact and encode the categorical variables in a one-hot manner, expanding to covariates, where n is the number of categorical variables, is the number of categories in each categorical variable i.

[0016] Preferably, the first data grouping module is further used to allocate the first subset of the sample batch to the experimental group, and allocate the second subset of the sample batch to the control group, and obtain an updated first Mahalanobis distance between the experimental group and the control group; The second data grouping module is further used to allocate the first subset of the sample batch to the control group, and allocate the second subset of the sample batch to the experimental group, and obtain the updated second Mahalanobis distance between the experimental group and the control group.

[0017] Preferably, the allocation control module is further used to control the sample batches to be allocated to the experimental group and the control group in a balanced manner according to the relationship between the first Mahalanobis distance and the second Mahalanobis distance.

[0018] Preferably, the proportional allocation control module is further used to obtain the relationship between the first Mahalanobis distance and the second Mahalanobis distance, including: if ,but ; if ,but ; if ,but ; Where 0.75<q<1, and the distribution To maintain equal proportion, i represents the i-th batch sample, 2i+1 and 2i+2 represent two subsets of the batch, M1 represents the first Mahalanobis distance, M2 represents the second Mahalanobis distance, and P() represents the probability of a given subset being assigned to the experimental grouping. ； When the number of sample batches is odd, the last sample batch is controlled to be allocated to the two groups with equal probability.

[0019] The present invention discloses the following technical effects: The present invention obtains more balanced grouping in algorithm experiments, and the evaluation and measurement of the business effects of specific algorithm actions are more accurate and less noisy, which is conducive to making more practical decisions; The present invention improves the efficiency of grouping, reduces the time cost of the experiment, and saves the calculation cost; The present invention can be directly deployed on the company's current infrastructure without the need for additional engineering modifications, thus saving the manpower cost of engineering modifications compared to the mbar solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 It is a schematic diagram of the method steps described in the present invention. DETAILED DESCRIPTION

[0022] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0023] like Figure 1-2 As shown, the present invention provides a rapid grouping method for AB test based on flow, which specifically includes the following steps: Step 1. Sample covariate processing: Numerical covariates remain unchanged, and categorical variables are encoded using one-hot encoding, expanding to covariates, where n is the number of categorical variables and m is the number of i is the number of categories in each categorical variable i; Step 2. Randomly arrange all N units (samples) into a sequence ; Step 3. Definition is the batch size, first and Assign groups, let T1 = 1 (experimental group) and T2 = 0 (control group). For simplicity, record , represents the sample of the i-th batch; Step 4. Assuming that 2i batches have been assigned groups, for the (2i+1)th and (2i+2)th batches: If the (2i+1)th batch is assigned to the experimental group and the (2i+2)th batch is assigned to the control group, then calculate the "potential" Mahalanobis distance between the updated experimental group and the control group ; Similarly, if the (2i+1)th batch is assigned to the control group and the (2i+2)th batch is assigned to the experimental group, another “potential” Mahalanobis distance is calculated: ; Step 5. Assign the (2i+1)th batch to the experimental group according to the following probability: if ,but ; if ,but ; if ,but ; Where 0.75<q<1, and the distribution To maintain equal proportions.

[0024] Step 6. Repeat the previous two steps until all units are assigned. is an odd number, the last batch is assigned to the two groups with equal probability. The flowchart is as follows Figure 1 shown.

[0025] Evaluation indicator establishment and experimental evaluation: 1. Use T test to perform significance test between groups for each covariate to confirm that the grouping is appropriate: For the same sample set, the original PSR method can give appropriate grouping for any arrangement. The improved batch PSR method needs to select the appropriate batch_size according to the size of the sample. Under the premise of appropriate selection, it can also achieve the goal of giving appropriate grouping for any arrangement. Under the premise of taking into account efficiency and availability, enlarging batch_size can further shorten the task time, and at least one usable result can be obtained in two partition attempts, and the total time consumption is lower than that of a single partition, that is, a suitable smaller batch_size configuration.

[0026] 2. Use a timer to calculate the program running time: The original psr algorithm uses Huawei Cloud 48 core VCores and 64G memory to group the driver task driver_psr, which takes an average of 4 hours. The driver group task driver_psr2 uses Huawei Cloud 16 core VCores and 32G memory, which takes an average of 8 hours. After the improved batch psr algorithm of the present invention, under the configuration of 16 core VCores and 32G memory, the above task execution takes an average of 8 minutes.

[0027] The present invention uses one-hot encoding to process categorical covariates; Based on the PSR algorithm, the present invention introduces batch processing of variables to be grouped, so that the program can group 2n variables instead of 2 variables at a time to speed up the grouping speed. When implementing the code, the present invention uses a generalized inverse matrix algorithm to calculate the inverse of the matrix, rather than a strict inverse matrix algorithm, when calculating the Mahalanobis distance, in order to ensure the robustness of the program implementation. The latter cannot invert a non-positive definite matrix. In the actual grouping process, the sample covariate matrix may be non-positive definite. The present invention adopts a suitable batch_size to achieve a balance between time and division effect according to the sample size of different division tasks.

[0028] The present invention supports balanced grouping under any number of covariate constraints, that is, for all given covariates, there is no statistical significance in the two divided groups; when processing the division of samples ranging from 100,000 to 1 million, the average time consumed by the task can reach 10 minutes.

[0029] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0030] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0031] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A rapid grouping method for AB test based on flow, characterized in that: The following steps are involved: After selecting samples for covariate expansion, they are randomly sorted to form a sequence consisting of several sample batches of the same size; Each sample batch of the same size is divided into two subsets of the same size, and the subsets are respectively assigned to the experimental group and the control group, and the sample batch corresponding to the current allocation progress is obtained; Allocating the first subset of the sample batch to the experimental group or the control group; assigning a second subset of the sample batch to a group different from the first subset; According to the Mahalanobis distance between the two allocation schemes of each allocated batch, the specific scheme for allocating the sample batch to the experimental group and the control group is determined to ensure that the data of the experimental group and the control group are balanced in the covariate dimension.

2. A rapid grouping method for AB test based on flow according to claim 1, characterized in that: In the process of expanding the sample covariates, the numerical covariates are kept as they are, and the categorical variables are encoded in a one-hot manner, expanding to covariates, where n is the number of categorical variables, is the number of categories in each categorical variable i.

3. A rapid grouping method for AB test based on flow according to claim 2, characterized in that: In the process of controlling the sample batch to be allocated to the experimental group and the control group for balanced allocation, a first subset of the sample batch is allocated to the experimental group, and a second subset of the sample batch is allocated to the control group, and a first Mahalanobis distance between the experimental group and the control group after the allocation is updated is obtained; Assigning a first subset of the sample batch to the control group, and assigning a second subset of the sample batch to the experimental group, to obtain an updated second Mahalanobis distance between the experimental group and the control group; According to the relationship between the first Mahalanobis distance and the second Mahalanobis distance, two subsets in the control sample batch are allocated to a specific scheme of balanced allocation between the experimental group and the control group.

4. A rapid grouping method for AB test based on flow according to claim 3, characterized in that: The allocation scheme for the two subsets of the sample batch is determined, and the corresponding relationship between the relative size of the first Mahalanobis distance and the second Mahalanobis distance is specifically expressed as: if ,but ; if ,but ; if ,but ; Where 0.75<q<1, distribution To ensure that the two subsets are not assigned to the experimental group or the control group at the same time, i represents the sample of the i-th batch, 2i+1 and 2i+2 represent the two subsets of the batch respectively, M1 represents the first Mahalanobis distance, M2 represents the second Mahalanobis distance, and P() represents the probability of assigning a given subset to the experimental group.

5. A rapid grouping method for AB test based on flow according to claim 4, characterized in that: In the process of controlling sample batches to be allocated to the experimental group and the control group in equal proportion, when the number of sample batches is an odd number, the last sample batch is controlled to be allocated to the two groups with equal probability.

6. A rapid grouping system for AB test based on flow, characterized in that: include: The data processing module is used to select samples for covariate expansion and then randomly sort them to form a sequence consisting of several sample batches of the same size; The data allocation module is used to allocate sample batches of the same size to the experimental group and the control group respectively, and obtain the sample batches corresponding to the current allocation progress; A first data grouping module, used for allocating the first subset of the sample batch to the experimental group, and allocating the second subset to the control group; A second data grouping module, used for allocating the first subset of the sample batch to the control group, and allocating the second subset to the experimental group; The allocation control module is used to control the allocation of the sample batches to the experimental group and the control group in a balanced manner according to the Mahalanobis distance between the first scheme of the allocated sample batches and the second scheme of the sample batches.

7. A rapid grouping system for flow-based AB test according to claim 6, characterized in that: The data processing module is also used to keep the numerical covariates intact and encode the categorical variables in a one-hot manner, which is expanded to covariates, where n is the number of categorical variables, is the number of categories in each categorical variable i.

8. A rapid grouping system for AB test based on flow according to claim 7, characterized in that: The first data grouping module is further used to allocate the first subset of the sample batch to the experimental group, and allocate the second subset of the sample batch to the control group, and obtain an updated first Mahalanobis distance between the experimental group and the control group; The second data grouping module is further used to allocate the first subset of the sample batch to the control group, and allocate the second subset of the sample batch to the experimental group, and obtain an updated second Mahalanobis distance between the experimental group and the control group.

9. A rapid grouping system for AB test based on flow according to claim 8, characterized in that: The allocation control module is further used to control the sample batches to be allocated to the experimental group and the control group in a balanced manner according to the relationship between the first Mahalanobis distance and the second Mahalanobis distance.

10. A rapid grouping system for flow-based AB test according to claim 9, characterized in that: The equal proportion distribution control module is also used to obtain the relationship between the first Mahalanobis distance and the second Mahalanobis distance, including: if ,but ; if ,but ; if ,but ; Where 0.75<q<1, and the distribution To maintain equal proportion, i represents the i-th batch of samples, 2i+1 and 2i+2 represent two subsets of the batch, M1 represents the first Mahalanobis distance, M2 represents the second Mahalanobis distance, and P() represents the probability of assigning a given subset to the experimental grouping; When the number of sample batches is odd, the last sample batch is controlled to be allocated to the two groups with equal probability.

Images