An AB experiment indicator management system and method based on user diversion

Abstract

The present invention belongs to the field of experimental management technology, and specifically relates to an AB experimental indicator management system and method based on user diversion. The method comprises: step 1: using the Flink real-time computing engine to obtain the behavioral data of users performing split tests in two or more variants; for each user, the behavioral intensity, behavioral participation depth, behavioral value and behavioral correlation are counted within a specified time window to form a feature vector; step 2: reducing the dimension of the feature vector to obtain the reduced-dimensional feature vector; using a stratified sampling method based on the reduced-dimensional feature vector to perform experimental grouping, and calculating the grouping probability of each user to construct a full user table of all users; step 3: in the full user table, calculating the atomic indicators and composite indicators of each experimental group. The present invention significantly improves the scientific nature of experimental design, the accuracy of experimental results and the efficiency of experimental management.

Description

Technical Field

[0001] The present invention belongs to the technical field of experiment management, and in particular relates to an AB experiment indicator management system and method based on user diversion. Background Art

[0002] With the rapid development of the Internet and mobile technology, enterprises are increasingly relying on data-driven decisions to optimize products and services and improve user experience and business benefits. As a widely used data analysis method, AB testing evaluates the impact of different variants (such as interface design, functional features, marketing strategies, etc.) on user behavior and business indicators by randomly assigning users to different experimental groups. It has become a standard tool for product optimization and business decision-making.

[0003] The existing AB testing methods mainly include the following categories:

[0004] Classic random grouping method: The simplest AB testing method, which randomly assigns users to the experimental group and the control group, and then compares the behaviors and indicators of different groups. Although this method is easy to implement, it is not effective when dealing with problems such as heterogeneity of user characteristics and data imbalance, which can easily lead to deviations in experimental results.

[0005] Stratified random grouping method: Based on the classic random grouping method, some characteristics of users (such as age, gender, region, etc.) are considered, and users are first stratified, and then randomly assigned within each layer. The stratified random grouping method solves the problem of user feature heterogeneity to a certain extent, but it still cannot handle the data imbalance problem caused by high-dimensional features and complex behavior patterns.

[0006] Feature-based grouping method: Use machine learning algorithms to group users based on their behavioral characteristics (such as historical purchase records, browsing behavior, etc.). This method can better capture users' complex behavioral patterns and improve the accuracy of experimental results. However, this method faces challenges in computational complexity, feature engineering, and model training, especially in terms of low efficiency when processing large-scale user data. Summary of the invention

[0007] In view of this, the main purpose of the present invention is to provide an AB experiment indicator management system and method based on user diversion, which solves the key problems in the existing AB testing methods through advanced data processing and analysis techniques, and significantly improves the scientific nature of the experimental design, the accuracy of the experimental results and the efficiency of the experimental management.

[0008] The technical solution adopted by the present invention is as follows:

[0009] An AB experiment indicator management system based on user diversion, the system comprises: a Flink real-time computing engine, which is used to obtain the behavior data of users in split tests in two or more variants; for each user, the behavior intensity, behavior participation depth, behavior value and behavior correlation are counted within a specified time window to form a feature vector; a user diversion part, which is used to reduce the dimension of the feature vector to obtain the reduced feature vector; based on the feature vector, a stratified sampling method is used to perform experimental grouping, and the grouping probability of each user is calculated to construct a full user table of all users; an experimental indicator calculation part, which is used to calculate the atomic indicator and composite indicator of each experimental group in the full user table.

[0010] A method for managing AB experiment indicators based on user diversion, the method comprising:

[0011] Step 1: Use the Flink real-time computing engine to obtain the behavior data of users in split tests in two or more variants. For each user, count their behavior intensity, behavior participation depth, behavior value, and behavior relevance within the specified time window to form a feature vector.

[0012] Step 2: Reduce the dimension of the feature vector to obtain the reduced feature vector; perform experimental grouping using the stratified sampling method based on the reduced feature vector, calculate the grouping probability of each user, and construct a full user table for all users;

[0013] Step 3: Calculate the atomic and composite indicators of each experimental group in the full user table.

[0014] Furthermore, in step 1, the feature vector is expressed using the following formula:

[0015] ;

[0016] in, is a feature vector, which contains four dimensions, namely, behavior intensity , Depth of Behavioral Engagement , Behavioral Value and behavioral relevance ; is the total number of behavior types; For the The number of times the behavior occurs; For the The cumulative duration of the behavior, in seconds; For the The intensity index of the behavior; is the preset novelty attenuation coefficient; For the The novelty score of the behavior; A constant to prevent division by zero; For the preset User engagement score for each behavior; is the preset time attenuation coefficient; is the current timestamp; For the The timestamp of the last time the behavior occurred; For the The deep engagement score of each behavior; is the average depth of engagement across all behaviors; For the preset The value score of the behavior; For the preset The importance weight of the behavior; are the parameters of the Sigmoid function, which are used to adjust the steepness and center point of the curve; For the The relevance score of the behavior; For the The degree of completion of the behavior; Parameters that affect the frequency of the preset adjustment behavior; For the The frequency of the behavior; is the average frequency of all behaviors.

[0017] Further, The intensity index of the behavior Calculated by the following formula:

[0018] ;

[0019] in, For the The average duration of the behavior; is the average duration of all behaviors; is the standard deviation of the duration of all behaviors; is the average number of steps of the 𝑖th behavior; is the average number of steps for all behaviors; is the standard deviation of the number of steps;

[0020] No. Novelty score of the behavior Calculated using the following formula:

[0021] ;

[0022] in, is the frequency of occurrence of the 𝑖th behavior among all users; is the frequency value corresponding to the behavior with the highest frequency among all behaviors.

[0023] Further, The deep engagement score of the behavior Calculated using the following formula:

[0024] ;

[0025] in, is the total number of steps of the 𝑖th behavior; is the user's first behavior The time spent on the step; is the user's first behavior The number of operations on the step; The degree of completion of this behavior is equal to The number of steps completed for a behavior divided by the The total number of steps of the behavior; The relevance score of the behavior It is equal to the number of other behaviors that occur immediately after the 𝑖th behavior divided by the total number of behaviors.

[0026] Furthermore, in step 2, the following formula is used to reduce the dimension of the feature vector to obtain the reduced dimension feature vector of:

[0027] ;

[0028] in, represents the t-distributed random neighbor embedding function; represents the uniform manifold approximation and projection function; Represents element-wise multiplication; is the feature vector The mean of is the feature vector The standard deviation of is a random noise term, which follows a normal distribution ; is a Gaussian kernel function that maps the feature vector to a high-dimensional space; is the Gaussian kernel matrix corresponding to the Gaussian kernel function; This is the transpose operation of a vector.

[0029] Furthermore, step 3 specifically includes: according to the feature vector after dimensionality reduction Divide users into several layers and use the K-means clustering algorithm to achieve stratification:

[0030] ;

[0031] in, It is clusters; It is The centers of the clusters; for each layer, calculate its weight:

[0032] ;

[0033] in It is The number of users in the layer; is the subscript index; experimental groups; for each layer, users are allocated according to the preset ratio of the experimental groups; For the The preset ratio of the experimental groups is Layer assigned to The number of users in the group is:

[0034] .

[0035] in, For the Layer assigned to The number of users in the group.

[0036] Furthermore, in step 2, the grouping probability of each user is calculated by the following formula: Users in Layer , which is assigned to The probability of a group is:

[0037] ;

[0038] in, Indicates experimental groups; Indicates user ; Is a user The reduced eigenvector of It is The center vector of each experimental group; is a preset adjustable smoothing parameter; a probability vector is constructed for each user through the following formula:

[0039] ;

[0040] The full user table is expressed using the following formula:

[0041] ;

[0042] in is the total number of users; For the The probability vector of each user; For the The reduced dimension feature vector of each user; Indicates The layer to which the user belongs.

[0043] Furthermore, in step 3, the atomic index is calculated using the following formula:

[0044] ;

[0045] in, It is Atomic index of experimental group The value of Indicates The layer is assigned to The number of users in the experimental group; For atomic index The standard value of atomic index, different atomic indexes correspond to different standard values ​​of atomic index;

[0046] Furthermore, in step 3, the composite index is calculated using the following formula:

[0047] ;

[0048] in, The experimental group Composite index The value of is the standard value of the composite index, and different composite indexes correspond to different composite index standard values; Atomic Index The weight of ; is the number of atomic indices participating in the compound; It is Expected assignment to treatment group in stratum Number of users; is an atomic index The average value among all experimental groups.

[0049] By adopting the above technical scheme, the present invention produces the following beneficial effects: the existing AB test method is often unable to cope with the heterogeneity of user characteristics, resulting in large deviations in experimental results. By introducing dimensionality reduction techniques such as Gaussian kernel function, t-SNE and UMAP, the present invention can effectively capture user behavior characteristics in high-dimensional space and map them to low-dimensional space, retaining important information while reducing computational complexity. This process ensures that the complexity and heterogeneity of user characteristics can be fully considered when performing user stratification and grouping, so that the distribution of user characteristics between experimental groups is more balanced, and the reliability of experimental results is significantly improved. The existing AB test method usually relies on a single indicator to evaluate the experimental effect, which is difficult to fully reflect the performance of the experimental group. The present invention proposes a set of scientific multi-indicator comprehensive evaluation methods, which comprehensively evaluate the performance of the experimental group by calculating atomic indicators and composite indicators. Atomic indicators combine factors such as user allocation probability, feature vector matching degree and user quantity deviation to ensure the accuracy and reliability of each indicator. Composite indicators integrate multiple atomic indicators, combine weights, logarithmic transformation and bias correction to form a comprehensive evaluation system that fully reflects the performance of the experimental group. This method can comprehensively evaluate the experimental effect in multiple dimensions, providing a solid data foundation for the analysis and decision-making of experimental results. In long-term experiments, user behavior and the external environment may change. Traditional AB testing methods are usually based on static designs and lack the ability to respond quickly to dynamic changes. The present invention can dynamically adjust the experimental design and provide real-time feedback by calculating the user's grouping probability and experimental indicators in real time. This capability makes experimental management more flexible and efficient, and can quickly respond to changes in user behavior and the external environment, ensuring the accuracy and timeliness of experimental results. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of an AB experiment indicator management method based on user diversion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0052] Embodiment 1: An AB experiment indicator management system based on user diversion, the system comprising: a Flink real-time computing engine, used to obtain the behavior data of users performing split tests in two or more variants; for each user, the behavior intensity, behavior participation depth, behavior value and behavior correlation are counted within a specified time window to form a feature vector; a user diversion part, used to reduce the dimension of the feature vector to obtain the reduced-dimensional feature vector; based on the feature vector, a stratified sampling method is used to perform experimental grouping, and the grouping probability of each user is calculated to construct a full user table of all users; an experimental indicator calculation part, used to calculate the atomic indicators and composite indicators of each experimental group in the full user table.

[0053] Embodiment 2: Based on the previous embodiment, refer to Figure 1 , a method for managing AB experiment indicators based on user diversion, the method comprising:

[0054] Step 1: Use the Flink real-time computing engine to obtain the behavior data of users in split tests in two or more variants. For each user, count their behavior intensity, behavior participation depth, behavior value, and behavior relevance within the specified time window to form a feature vector.

[0055] Flink's real-time computing engine can process large amounts of streaming data, ensuring that user behavior data can be quickly acquired and processed during the test, thereby achieving real-time monitoring and data analysis. In order to better understand user behavior and conduct effective experimental grouping, it is necessary to count the behavior intensity, behavior participation depth, behavior value, and behavior relevance of each user within a specified time window, and combine these statistics into a feature vector. These feature vectors will be used for subsequent dimensionality reduction and experimental grouping operations. Behavior intensity refers to the frequency and degree of user interaction with the experimental variant over a period of time, which can be quantified by counting specific behavioral indicators such as the number of clicks and page dwell time of the user. For example, the more times a user clicks in a specific time period, the higher the behavior intensity, which reflects the user's attention and interaction with a certain variant. Behavior participation depth describes the degree of user involvement in the experimental variant, which is usually measured by the level of pages visited by the user. If a user browses multiple pages in depth in a variant, the depth of participation is high, which can indicate the user's interest in and willingness to explore the content of the variant. Behavior value is the actual value brought by evaluating user behavior, such as the revenue generated by the user in the variant and the purchases completed. High-value behavior indicates that the variant is attractive and convertible to users, which is one of the important criteria for the success or failure of the experiment. Behavior relevance refers to the degree of correlation between user behavior and the experimental goal. This indicator can be evaluated by the correlation between user behavior and the key performance indicators (KPIs) of the experiment. The construction of feature vectors not only improves the efficiency of data processing, but also effectively avoids data redundancy and information loss. In this step, the feature vector is not just a simple collection of multiple behavioral indicators, but also a process of extracting key features through comprehensive analysis of user behavior. This method can better capture the diversity and complexity of user behavior and provide guarantees for the accuracy and scientificity of experimental results. By analyzing the feature vectors, the user's behavior patterns in different variants can be discovered, which provides an important reference for subsequent experimental grouping and result analysis. In the specific implementation, the Flink engine can perform real-time calculation and aggregation of a large amount of user behavior data through its stream processing framework. For example, when a user switches between variant A and variant B, the Flink engine can capture this behavior in real time and count and update its behavior intensity, participation depth, behavior value and correlation data. By setting an appropriate time window, the timeliness and accuracy of statistical data can be ensured, so that the feature vector can accurately reflect the latest user behavior dynamics. This process not only improves the real-time nature of data processing, but also effectively improves the efficiency and accuracy of AB testing.

[0056] Step 2: Reduce the dimension of the feature vector to obtain the reduced feature vector; perform experimental grouping using the stratified sampling method based on the reduced feature vector, calculate the grouping probability of each user, and construct a full user table for all users;

[0057] Eigenvector dimensionality reduction is an important technology in data processing. Its purpose is to reduce the number of features while maintaining the main information of the data, thereby reducing the computational complexity and avoiding overfitting. High-dimensional feature data often faces problems such as large computational complexity and high noise interference when processing. To overcome these challenges, this method introduces dimensionality reduction techniques such as principal component analysis (PCA) and linear discriminant analysis (LDA). These methods project high-dimensional data into low-dimensional space through mathematical transformation to extract the most representative features. To implement the dimensionality reduction process, the eigenvectors need to be centered first, that is, the mean is subtracted to concentrate the data distribution. Next, the covariance matrix of the eigenvector is calculated, which reflects the correlation between the features. The covariance matrix is ​​then eigenvalue decomposed to obtain eigenvalues ​​and eigenvectors. The eigenvalue represents the variance of the data in the direction of the corresponding eigenvector, and the larger eigenvalue represents the main direction of change of the data. Selecting the eigenvectors corresponding to the first few largest eigenvalues ​​as the basis vectors of the new feature space can effectively reduce the feature dimension. In the new feature space, the original eigenvector is projected into the eigenvector after dimensionality reduction. This process simplifies the data structure and improves computational efficiency by extracting the main features of the data, while retaining the main information of the data. The feature vector after dimensionality reduction provides the basis for subsequent stratified sampling and experimental grouping. Stratified sampling is an effective method to ensure the balance of features between experimental groups. Through stratified sampling, we can ensure that the distribution of user features in different experimental groups is similar, avoid grouping bias, and improve the credibility of experimental results. In specific operations, we first stratify users based on the feature vector after dimensionality reduction, and each layer represents a user group with similar features. Within each layer, random sampling is performed according to the predetermined grouping probability to assign users to different experimental groups.

[0058] For example, suppose we use PCA to reduce the dimension and obtain the feature vector after the dimension reduction ,in and are two main features. When stratifying users, users can be divided into several layers based on these two features. Users in each layer have similar and value. Then, in each layer, random sampling is performed according to a predetermined grouping probability, such as 50%, and half of the users are assigned to experimental group A and the other half to experimental group B. In this way, it can be ensured that experimental group A and experimental group B have similar user feature distributions as a whole, thereby improving the reliability of the experimental results. The full user table is the final product of experimental grouping, which records the grouping information and grouping probability of each user. The full user table not only contains the basic information and behavioral characteristics of the user, but also includes its distribution in different experimental groups. This table provides basic data for the subsequent calculation of experimental indicators. When calculating experimental indicators, different experimental groups can be compared and analyzed based on the information in the full user table to evaluate the effects of each experimental variant. For example, when calculating atomic indicators, the click-through rate, conversion rate and other behavioral data of each user in experimental group A and experimental group B can be counted. By comparing these data, the impact of different experimental variants on user behavior can be evaluated. When calculating composite indicators, multiple atomic indicators can be weighted averaged or combined in other forms to obtain more comprehensive evaluation indicators, such as customer lifetime value (LTV) and net promoter score (NPS). The detailed grouping information and behavior data provided by the full user table make the calculation of these indicators more accurate and comprehensive. In the method of the present invention, dimensionality reduction and stratified sampling are key steps to ensure the scientificity of experimental grouping and the accuracy of results. Through dimensionality reduction technology, we can simplify the data structure, improve calculation efficiency, and retain the main information of the data. Through the stratified sampling method, we can ensure that the user characteristics between different experimental groups are evenly distributed, avoid grouping bias, and improve the reliability and scientificity of the experimental results. The construction of the full user table provides a solid data foundation for the subsequent calculation of experimental indicators. Specifically, dimensionality reduction technology reduces the complexity of data processing by reducing the feature dimension, while retaining the main information of the data, making subsequent grouping and analysis more efficient. The stratified sampling method ensures the balance of features between the experimental groups by stratifying the users, avoiding the deviation of experimental results caused by the imbalance of user characteristics. The construction of the full user table integrates the grouping information and behavior data of all users, providing detailed data support for the subsequent calculation of experimental indicators. In summary, step 2 optimizes the process of experimental grouping through the method of feature vector dimensionality reduction and stratified sampling, and provides technical guarantee for the realization of scientific and effective AB experiments. Dimensionality reduction technology simplifies the data structure and improves computational efficiency. The stratified sampling method ensures the scientific nature of the experimental grouping and the reliability of the results. The construction of the full user table provides a solid data foundation for the calculation of experimental indicators. These technological innovations and method improvements make the method of the present invention have significant advantages in AB experimental indicator management, and can perform experimental grouping and data analysis more scientifically and efficiently, thereby achieving more accurate experimental results and more effective experimental management.

[0059] Step 3: Calculate the atomic and composite indicators of each experimental group in the full user table.

[0060] In the full user table, the grouping information and behavior data of all users have been fully recorded. The full user table contains all the key behavior data of each user during the experiment, which is the basis for calculating various indicators. Atomic indicators are statistical data of a single behavior or event. Common atomic indicators include click-through rate, conversion rate, page dwell time, etc. These indicators directly reflect the basic behavioral characteristics of users in different experimental groups. For example, click-through rate can measure the user's interest in a certain variant, while conversion rate can evaluate the actual effect of the variant and user conversion. When calculating atomic indicators, it is necessary to summarize and count the user behavior data in each experimental group to ensure the accuracy and reliability of the statistical results. Composite indicators are comprehensive evaluation indicators composed of multiple atomic indicators, which are usually used to more comprehensively evaluate the experimental effect. The design of composite indicators needs to take into account the specific goals and evaluation criteria of the experiment. Common composite indicators include customer lifetime value (LTV), net promoter score (NPS), user retention rate, etc. For example, customer lifetime value (LTV) can be calculated by combining the user's purchase behavior during the experiment and the expected future purchase behavior. This indicator can help evaluate the impact of experimental variants on the long-term value of users. The Net Promoter Score (NPS) reflects the overall satisfaction and loyalty of users to the variants by calculating the difference in users' willingness to recommend. The user retention rate evaluates the impact of the variant on user stickiness by counting the continuous activity of users during the experiment. When calculating these indicators, it is necessary to carefully process and analyze the data in the full user table. First, for atomic indicators, it is necessary to extract the behavioral data of each user according to the experimental design and data collection plan. For example, for click-through rate, the number of clicks of each user during the experiment can be counted, and the average click-through rate of the experimental group can be calculated. Similarly, for conversion rate, the conversion behavior of each user during the experiment, such as purchase, registration, etc., can be counted, and the average conversion rate of the experimental group can be calculated. Through these atomic indicators, the basic behavioral characteristics and effect differences of each experimental group can be preliminarily evaluated. For the calculation of composite indicators, more complex data processing and analysis methods are required. For example, when calculating customer lifetime value (LTV), it is necessary not only to consider the user's purchase behavior during the experiment, but also to combine the user's historical purchase data and expected future purchase behavior. The user behavior prediction model can be used to predict the user's future purchasing behavior, and the purchasing behavior during the experiment can be combined with the prediction results to calculate the customer lifetime value. The calculation of the Net Promoter Score (NPS) requires statistical distribution of users' willingness to recommend variants through user surveys or feedback data, and calculates the difference between the proportion of recommenders and the proportion of detractors. The calculation of user retention rate requires statistical analysis of the number of days that users are continuously active during the experiment to evaluate the impact of variants on user stickiness. In the entire indicator calculation process, the accuracy of the data and the rationality of the processing method are crucial.To ensure the reliability of the calculation results, the data needs to be preprocessed and cleaned to remove outliers and noise data. At the same time, appropriate statistical and analytical methods are used to ensure the scientificity and fairness of indicator calculation. Through the comprehensive analysis of atomic indicators and composite indicators, the effects of each experimental group can be comprehensively evaluated, providing strong support for the decision-making of experimental results. In addition, in order to more intuitively display the experimental results, the calculation results of various indicators can be visualized. Through charts, dashboards, etc., the indicators of different experimental groups are displayed for easy comparison and analysis. For example, a bar chart of click-through rate and conversion rate can be drawn to show the basic behavioral characteristics of each experimental group; a line chart of customer lifetime value (LTV) and net recommendation value (NPS) can be drawn to show the comprehensive evaluation results of each experimental group; and a curve chart of user retention rate can be used to evaluate the changes in user stickiness of each experimental group. Through these visualization tools, the experimental results can be understood and analyzed more intuitively, and potential problems and improvement directions can be discovered.

[0061] Embodiment 3: Based on the previous embodiment, in step 1, the feature vector is represented by the following formula:

[0062] ;

[0063] in, is a feature vector, which contains four dimensions, namely, behavior intensity , Depth of Behavioral Engagement , Behavioral Value and behavioral relevance ; is the total number of behavior types; For the The number of times the behavior occurs; For the The cumulative duration of the behavior, in seconds; For the The intensity index of the behavior; is the preset novelty attenuation coefficient; For the The novelty score of the behavior; A constant to prevent division by zero; For the preset User engagement score for each behavior; is the preset time attenuation coefficient; is the current timestamp; For the The timestamp of the last time the behavior occurred; For the The deep engagement score of each behavior; is the average depth of engagement across all behaviors; For the preset The value score of the behavior; For the preset The importance weight of the behavior; are the parameters of the Sigmoid function, which are used to adjust the steepness and center point of the curve; For the The relevance score of the behavior; For the The degree of completion of the behavior; Parameters that affect the frequency of the preset adjustment behavior; For the The frequency of the behavior; is the average frequency of all behaviors.

[0064] Specific, behavioral intensity The calculation formula combines the number of user behaviors , the cumulative duration of the behavior , Behavior Intensity Index and behavioral novelty scores The formula is designed to take into account the normalization of the number of behaviors, by dividing by the maximum value To eliminate the dimensional differences between different behavior times. This step ensures the comparability of the behavior data. The logarithmic transformation is introduced in the formula , used to smooth the duration of the behavior The logarithmic transformation can alleviate the excessive impact of long-term behavior on the results to a certain extent, so that short-term behavior can also be reasonably reflected. Behavior intensity index It directly measures the importance of the behavior. Different behaviors can be assigned different strength indexes to reflect their weight in the overall experiment. and novelty decay coefficient The introduction of is to reflect the impact of the novelty of behavior on the intensity of user behavior. The more novel the behavior is, the greater the weight in the intensity calculation, which is expressed by This part ensures that the novel behavior is important in the experiment. The sum of squares of the denominator and the constant added to it The operation is to prevent division by zero errors and ensure the stability of calculations and the rationality of results. The calculation formula takes into account the depth of user engagement behavior and the time decay factor. User Engagement Score It is an important indicator to measure user participation behavior. Time decay coefficient and the timestamp of the last time the behavior occurred Through the exponential decay model To adjust the engagement score, the most recent behavior has a greater impact on the engagement depth. This design ensures that recent behavior is important in the engagement depth calculation. Introduced to handle deep engagement scores for behaviors . Average level of engagement relative to all behaviors , the hyperbolic tangent function can smoothly handle the changes in the depth of engagement, making the calculation results more stable and reasonable. The exponential decay and summation operations in the denominator ensure the comprehensive impact of behaviors at different time points on the results. The weight of behaviors that occur earlier gradually decreases, thus reflecting the dynamic changes in user behavior participation. Behavior Value The calculation formula involves the value score of the behavior and importance weights The formula calculates the comprehensive value of the behavior through weighted average and the Sigmoid function parameter and Adjust the steepness of the curve and the position of the center point. Specifically, the design of the formula ensures that the importance and actual value of the behavior can be comprehensively considered during the calculation process, so that high-value behaviors occupy a larger proportion in the results. The introduction of the Sigmoid function makes the calculation results of the behavior value more sensitive within a certain range, thereby better reflecting the fluctuations of the behavior value. Negative signs and parameters Used to adjust the range and sensitivity of the result, parameters Used to adjust the baseline value so that the formula can be flexible when dealing with different types of behavior values. Through this design, the calculation of behavior value can accurately reflect the relative importance and comprehensive value of each behavior in the experiment. Behavioral relevance The calculation formula combines the behavioral relevance score , Behavior completion and behavior frequency Through logarithmic transformation and Sigmoid function, the formula can smooth the impact of completion and frequency, so that the relevance score more accurately reflects the actual situation of user behavior. Specifically, Processing completion ensures the nonlinear effect of completion and prevents low completion behavior from having an excessive impact on the results. Sigmoid function Describing the frequency of behavior Relative to the average frequency The higher the frequency, the greater the weight. This design ensures the importance of high-frequency behavior in correlation calculation, while preventing behaviors with too low frequency from affecting the accuracy of the results. In the entire process of constructing the feature vector, the calculation of each dimension introduces reasonable parameters and mathematical models to ensure that different aspects of user behavior can be accurately reflected. In the behavior intensity formula, normalization and logarithmic transformation ensure the comparability and stability of behavior data; in the behavior participation depth formula, the exponential decay model and hyperbolic tangent function smooth the influence of time and depth; in the behavior value formula, the weight and Sigmoid function ensure the sensitivity and accuracy of value calculation; in the behavior correlation formula, the logarithmic transformation and Sigmoid function smooth the influence of completion and frequency. In this way, the constructed feature vector not only fully reflects the behavioral characteristics of users in different variants, but also provides a solid data foundation for subsequent dimensionality reduction and experimental grouping. In the process of dimensionality reduction, the feature vector will be further simplified, making data processing more efficient; in the process of experimental grouping, the stratified sampling method ensures the balanced distribution of user characteristics between different experimental groups, thereby improving the reliability and scientificity of the experimental results.

[0065] Embodiment 4: Based on the previous embodiment, The intensity index of the behavior Calculated by the following formula:

[0066] ;

[0067] in, For the The average duration of the behavior; is the average duration of all behaviors; is the standard deviation of the duration of all behaviors; is the average number of steps of the 𝑖th behavior; is the average number of steps for all behaviors; is the standard deviation of the number of steps;

[0068] No. Novelty score of the behavior Calculated using the following formula:

[0069] ;

[0070] in, is the frequency of occurrence of the 𝑖th behavior among all users; is the frequency value corresponding to the behavior with the highest frequency among all behaviors.

[0071] Specifically, the Behavior Intensity Index The calculation formula uses the Sigmoid function to convert the average duration of the behavior and the average number of steps After normalization, it is mapped to the range between 0 and 1. This processing method has good numerical stability and interpretability. Specifically, the formula is transformed into The average duration of the behavior The average duration of all behaviors Compare and use the standard deviation of all behavior durations Similarly, The average number of steps in a behavior Also the average number of steps for all behaviors For comparison, use the standard deviation of the number of steps This process eliminates the dimensional differences between different behaviors, allowing the characteristics of each behavior to be compared on a unified scale. By converting the standardized duration and the sum of the number of steps to a Sigmoid function, the formula ensures that the behavior intensity index It always changes between 0 and 1. This design not only ensures the numerical stability of the results, but also effectively distinguishes behaviors of different intensities. On the other hand, the behavior novelty score The calculation formula of evaluates the novelty of behavior by combining the frequency of behavior occurrence and the time decay factor. Specifically, the formula first calculates the The frequency of this behavior among all users , and calculate the ratio of its logarithm to the logarithm of the behavior with the highest frequency among all behaviors. This processing method ensures that the novelty score of behaviors with higher frequency is lower, and the novelty score of behaviors with lower frequency is higher. At the same time, the formula introduces the time decay coefficient and the timestamp of the last time the behavior occurred , through the exponential decay model Adjust the novelty score so that the most recent behavior has a greater impact on the novelty score. This design takes into account the time dimension of the behavior and ensures that the novelty score can dynamically reflect the timeliness of the behavior. These calculation methods are of great significance in the management of AB experiment indicators. By fully quantifying the intensity and novelty of the behavior, the behavioral characteristics of users in different variants can be more accurately captured, thereby providing a high-quality data basis for experimental grouping and result analysis. Behavior Intensity Index The design enables the experiment to distinguish users’ behavioral responses of different strengths, thereby evaluating the attractiveness and interaction effect of the variants. It helps identify user responses to novel and rare behaviors, thereby evaluating the effectiveness of variants in inducing new behaviors.

[0072] Embodiment 5: Based on the previous embodiment, The deep engagement score of the behavior Calculated using the following formula:

[0073] ;

[0074] in, is the total number of steps of the 𝑖th behavior; is the user's first behavior The time spent on the step; is the user's first behavior The number of operations on the step; The degree of completion of this behavior is equal to The number of steps completed for a behavior divided by the The total number of steps of the behavior; The relevance score of the behavior It is equal to the number of other behaviors that occur immediately after the 𝑖th behavior divided by the total number of behaviors.

[0075] Specifically, behavioral deep engagement score The calculation formula is designed to measure the depth of user engagement in a specific behavior. The formula takes into account the time and number of operations a user takes on each step to provide a comprehensive engagement score. Specifically, the formula evaluates the depth of user engagement in a behavior by taking a weighted average of the time spent on each step. It represents the decreasing participation of users in the subsequent steps of the behavior, ensuring that the initial steps of the behavior have a greater impact, while the impact of the subsequent steps gradually weakens. This weighting method can reflect the high participation of users in the early stages of the behavior and their gradual withdrawal in the later stages. At the same time, the formula also uses logarithmic transformation Considers the number of operations the user performs on each step . The logarithmic transformation of the number of operations allows the frequent operation of the behavior to be properly reflected without causing the result to be biased by extremely high-frequency operations. In this way, the behavioral deep engagement score not only takes into account the time investment, but also integrates the frequency of operations, thereby providing a more comprehensive measure of engagement. This design is of great significance in the management of AB experiment indicators. By accurately quantifying the depth of user behavior, we can better understand the specific ways in which users interact in different variants. This is crucial for evaluating the attractiveness of variants and the user engagement effect. For example, if a variant can significantly improve the depth of user engagement, it means that the variant is more attractive in design and content, and can more effectively attract users to invest time and energy. On the other hand, the behavioral relevance score The calculation formula evaluates the relevance of a behavior by measuring the frequency of other behaviors that occur immediately after a certain behavior. The design idea of ​​this formula is to judge the importance of a certain behavior in the overall behavior sequence of the user by analyzing the coherence and correlation between behaviors. If a certain behavior is followed by a large number of other behaviors, it indicates that the behavior has a high relevance and importance in the user's behavior path. In the specific implementation, the behavior relevance score is calculated by counting the number of behaviors that occur immediately after the first The number of other behaviors that follow a behavior is compared with the total number of behaviors to obtain a relative frequency. This relative frequency reflects the strength of the association between behaviors, allowing the experimental design to identify which behaviors have an important impact on the user's overall behavior path. This analysis method is also crucial in AB experiments. By understanding the correlation between behaviors, the experimental design can be optimized, key behavior nodes can be identified, and targeted optimization can be performed. For example, if a certain behavior is found to have a high correlation, its role in the user's decision-making process can be further analyzed, so that this behavior can be highlighted in the variant design to improve the overall experimental effect.

[0076] Example 6: Based on the previous example, the following formula is used in step 2 to reduce the dimension of the feature vector to obtain the reduced dimension feature vector of:

[0077] ;

[0078] in, represents the t-distributed random neighbor embedding function; represents the uniform manifold approximation and projection function; Represents element-wise multiplication; is the feature vector The mean of is the feature vector The standard deviation of is a random noise term, which follows a normal distribution ; is a Gaussian kernel function that maps the feature vector to a high-dimensional space; is the Gaussian kernel matrix corresponding to the Gaussian kernel function; This is the transpose operation of a vector.

[0079] Specifically, in the process of feature vector dimensionality reduction, the Gaussian kernel function is used The original high-dimensional feature vector Mapped to a higher dimensional space. The role of the Gaussian kernel function is to capture the nonlinear relationship in the feature vector, so that the feature vector can better separate the features of different categories in the high-dimensional space. Specifically, the Gaussian kernel function calculates the feature vector and its mean The square of the difference and divided by the standard deviation of the eigenvector , and then substitute it into the exponential function to achieve the mapping of high-dimensional space. In this way, the Gaussian kernel function can convert complex nonlinear relationships into linear relationships in high-dimensional space, making the subsequent dimensionality reduction processing more effective. After completing the Gaussian kernel function mapping, the t-SNE method is used to perform preliminary dimensionality reduction on the feature vectors in the high-dimensional space. t-SNE is a nonlinear dimensionality reduction technique that aims to embed high-dimensional data into low-dimensional space by retaining the local proximity relationship between data points. In this method, t-SNE calculates the similarity between feature vectors in high-dimensional space and maps this similarity to low-dimensional space, so that similar data points maintain a closer distance in low-dimensional space. The core idea of ​​t-SNE is to ensure that the feature vector after dimensionality reduction can retain the local structure and relationship of the original data by minimizing the difference in distribution between data points in high-dimensional space and low-dimensional space. This process enables the data to still accurately reflect the relationship between the original features after dimensionality reduction, providing an accurate data representation. However, t-SNE focuses on the local structure of the data during dimensionality reduction, and may ignore the information of the global structure. Therefore, in order to further improve the dimensionality reduction effect and retain the global structure of the data, this method introduces the UMAP method for further processing after t-SNE dimensionality reduction. UMAP is a new nonlinear dimensionality reduction method that not only focuses on the local proximity relationship of data points, but also preserves the global structure of the data by constructing a topological graph in a high-dimensional space and embedding it into a low-dimensional space. UMAP constructs and optimizes a proximity graph and then projects it in a low-dimensional space, so that the feature vector after dimensionality reduction can simultaneously reflect the local and global characteristics of the data. This processing method ensures that the representation of the data after dimensionality reduction is more complete and accurate.

[0080] In the UMAP dimensionality reduction process, UMAP can effectively capture the global pattern of the data by constructing a topological graph of the data points in the high-dimensional space and mapping this structure to the low-dimensional space. This method combines the concepts of graph theory and geometry, and optimizes the edge weights in the neighboring graph so that the arrangement of data points in the low-dimensional space can retain the structural characteristics in the high-dimensional space. In this way, UMAP can not only provide an accurate local structural representation, but also ensure the fidelity of the global structure, thereby retaining more original information in the feature vector after dimensionality reduction. In order to further enhance the stability and accuracy of the dimensionality reduction results, this method also introduces a random noise term , which follows a normal distribution The introduction of random noise is intended to prevent overfitting and ensure that the feature vectors after dimensionality reduction have good separability and representativeness between different experimental groups. By adding an appropriate amount of random noise during the dimensionality reduction process, the robustness of the model can be improved, making the dimensionality reduction results more robust and reliable. In the entire dimensionality reduction process, the element-by-element multiplication operation It is used to combine the mapping results of the Gaussian kernel function with the dimensionality reduction results of t-SNE and UMAP, so that the final reduced-dimensional feature vector is smoother and more stable. This operation ensures that the representation of the feature vector after dimensionality reduction not only retains the main information of the original data, but also effectively reduces the impact of noise and outliers, and improves the overall quality of data processing. Through the above steps, the feature vector obtained after dimensionality reduction is It has the following characteristics: First, through the mapping of Gaussian kernel function, it can capture the nonlinear relationship in the feature vector, making the representation in high-dimensional space more accurate; second, through the joint application of t-SNE and UMAP, it can retain the local and global structure of the data in the low-dimensional space, ensuring that the feature vector after dimensionality reduction can accurately reflect the characteristics of the original data; finally, by introducing random noise terms and element-by-element multiplication operations, the stability and separability of the dimensionality reduction results are ensured. These characteristics make this method have significant advantages in data processing and analysis. In practical applications, this dimensionality reduction method can significantly improve the efficiency of AB experiments and the accuracy of results. By reducing the dimensionality of the feature vector, the computational complexity can be effectively reduced and the data processing speed can be improved. At the same time, by retaining the main information of the feature vector, it is ensured that the data after dimensionality reduction can accurately reflect the user behavior characteristics, thereby providing high-quality data support for experimental grouping and result analysis. Especially in the scenarios of large-scale data analysis and high-dimensional data processing, this dimensionality reduction method can significantly improve the analysis efficiency and reduce the consumption of computing resources, thereby better supporting business decision-making and product optimization.

[0081] Embodiment 7: Based on the previous embodiment, step 3 specifically includes: according to the feature vector after dimensionality reduction Divide users into several layers and use the K-means clustering algorithm to achieve stratification:

[0082] ;

[0083] in, It is clusters; It is The centers of the clusters; for each layer, calculate its weight:

[0084] ;

[0085] in It is The number of users in the layer; is the subscript index; experimental groups; for each layer, users are allocated according to the preset ratio of the experimental groups; For the The preset ratio of the experimental groups is Layer assigned to The number of users in the group is:

[0086] .

[0087] in, For the Layer assigned to The number of users in the group.

[0088] Specifically, the K-means clustering algorithm is used to stratify the feature vector after dimension reduction. K-means is a commonly used unsupervised learning algorithm, whose goal is to divide data points into The sum of the distances between the data points in each cluster and the center of the cluster is minimized. Specifically, K-means achieves this goal by iteratively optimizing the objective function: the objective function is in the form of the within-cluster sum of squares (WCSS), which is the sum of the squares of the distances from all data points to the center of the cluster to which they belong. This optimization process repeatedly adjusts the center point of each cluster. , so that each data point The distance to its nearest center point is the smallest, thus achieving the best clustering effect. After clustering is completed, we get clusters, each cluster contains several users. , calculate its weight , that is, the proportion of users in this cluster to the total number of users. The weight is calculated by dividing the number of users in each cluster by the sum of the number of users in all clusters. This step ensures that in the subsequent user allocation process, users can be reasonably allocated to different experimental groups according to the size of each cluster. Next, according to the preset proportion of each experimental group , assign the users in each cluster to different experimental groups. The specific assignment process is: for each cluster , according to the preset ratio and cluster weight , calculate the number of users that should be assigned to each experimental group. The rounding function is used in the calculation formula for the number of assigned users , to ensure that the number of users allocated is an integer. In this way, the users in each cluster will be evenly allocated to different experimental groups according to the preset ratio and cluster weight, so as to ensure that the distribution of user characteristics between the experimental groups is as consistent as possible. This stratification and allocation method is of great significance in AB experiments. First, stratification through the K-means clustering algorithm can effectively group users according to their feature vectors, so that the user characteristics in each cluster are similar, so that when users are allocated to experimental groups, the user characteristics of each experimental group can be evenly distributed. Secondly, by calculating the cluster weights and allocating users according to the preset ratio, the reasonable distribution of the number of users in different experimental groups can be ensured, avoiding the deviation of experimental results caused by uneven user distribution. This method not only improves the reliability of the experimental results, but also more accurately reflects the impact of different variants on user behavior. In practical applications, this user stratification method based on K-means clustering and preset ratio allocation can significantly improve the scientificity and effectiveness of AB experiments. For example, in an AB test of a website, if you want to compare the impact of two different page designs on user click-through rates, you can first reduce the dimension of the feature vector to obtain the feature representation of each user, and then use the K-means algorithm to divide the users into several clusters and assign the users to different page versions according to the preset ratio. This method ensures that the user feature distribution in each page version is similar, thereby eliminating the impact of user feature differences on the experimental results, making the experimental results more credible. In addition, this method has good scalability and flexibility. By adjusting the number of clusters of the K-means algorithm and the preset ratio of the experimental group , which can flexibly adapt to the experimental requirements of different scales and complexities. At the same time, the combination of the feature vector after dimensionality reduction and the K-means clustering algorithm also provides an efficient solution for processing large-scale data.

[0089] Embodiment 8: Based on the previous embodiment, in step 2, the grouping probability of each user is calculated by the following formula: Users in Layer , which is assigned to The probability of a group is:

[0090] ;

[0091] in, Indicates experimental groups; Indicates user ; Is a user The reduced eigenvector of It is The center vector of each experimental group; is a preset adjustable smoothing parameter; a probability vector is constructed for each user through the following formula:

[0092] ;

[0093] The full user table is expressed using the following formula:

[0094] ;

[0095] in is the total number of users; For the The probability vector of each user; For the The reduced dimension feature vector of each user; Indicates The layer to which the user belongs.

[0096] Specifically, for Users in Layer , which is assigned to Probability of group It consists of two main parts. The first part Indicates Users in the layer are assigned to The initial proportion of the group. Here, It is The layer is assigned to The number of users in the group, It is The total number of users in the layer. This part reflects the preset user allocation ratio of each experimental group in each layer. The second part is a probability density function based on Gaussian distribution. Here, Is a user The reduced eigenvector of It is The center vector of each experimental group, is a preset adjustable smoothing parameter. This part represents the similarity between the user feature vector and the center vector of each experimental group. By calculating the Euclidean distance between the user feature vector and the center vector of each experimental group and substituting it into the probability density function of the Gaussian distribution, the relative weight of each experimental group is obtained. Then, these weights are normalized by the Softmax function to ensure that the total probability is 1. Specifically, the probability density function part of the Gaussian distribution Describes the degree of match between the user feature vector and the center vector of the experimental group. The closer the distance, the higher the degree of match and the greater the probability. This method ensures that users can be reasonably assigned to the experimental group that best matches their characteristics by quantifying the similarity between user features and experimental group features. By combining the above two parts, the grouping probability of each user is finally calculated. This probability not only takes into account the preset allocation ratio of each experimental group, but also combines the degree of match between user features and experimental group features, thereby achieving a more scientific and reasonable user grouping. Next, construct a probability vector for each user , contains the probability of the user being assigned to each experimental group. Specifically, the probability vector Each element of Indicates user Assigned to The probability of being assigned to an experimental group. This probability vector provides basic data for the subsequent construction of the full user table. The full user table contains the reduced feature vector of each user, the layer to which the user belongs, and the probability of the user being assigned to each experimental group. In this way, the grouping information of each user can be comprehensively recorded and managed, so that the behavioral data of each user can be accurately tracked and analyzed during the experiment. This method of calculating the probability of grouping each user is of great significance in AB experiments. First, by combining the matching degree of user characteristics with the experimental group characteristics, it can ensure that each user is assigned to the experimental group that best matches his or her characteristics, thereby improving the accuracy and reliability of the experimental results. Secondly, by introducing Gaussian distribution and Softmax function, the complexity and diversity of user characteristics can be effectively handled, ensuring the scientificity and rationality of the grouping process. In practical applications, this probability-based user grouping method can significantly improve the effect of AB experiments. For example, in the AB test of an e-commerce website, this method can be used to reasonably assign users to different product recommendation algorithm experimental groups, so as to more accurately evaluate the effects of each algorithm. By recording and analyzing the data in the full user table, the behavioral differences of users in different experimental groups can be fully understood, thus providing strong support for product optimization and decision-making.

[0097] Embodiment 9: Based on the previous embodiment, in step 3, the atomic index is calculated using the following formula:

[0098] ;

[0099] in, It is Atomic index of experimental group The value of Indicates The layer is assigned to The number of users in the experimental group; For atomic index The standard value of atomic index, different atomic indexes correspond to different standard values ​​of atomic index.

[0100] Specifically, atomic indicators The calculation first combines the user's assigned The probability of the experimental group and the matching degree of the feature vector with the center vector of the experimental group. Specifically, the user Assigned to The probability of an experimental group Indicates the importance of the user in the experiment allocation. The similarity between the user feature vector and the experimental group center vector is quantified. By combining these two parts, the numerator of the formula The contribution and feature matching of each user in the experiment are comprehensively considered to obtain the weighted sum of the feature matching of the entire experimental group. Is for all users assigned to This step is used to normalize the calculation results of the molecular part to ensure that the final atomic index values ​​are comparable and stable. Through this normalization process, the influence caused by the difference in the number of users in different experimental groups can be eliminated, so that the atomic indexes of each experimental group can be compared under the same standard. In order to further calibrate the calculation results of the atomic index, the formula is introduced , that is, the atomic index The standard value of atomic index. There may be different standard values, which are used to calibrate each atomic indicator so that it has a unified dimension in the calculation, thereby ensuring the comparability between different atomic indicators. For example, different types of atomic indicators such as click-through rate and conversion rate may require different standard values ​​for unified calibration. In this way, the results of atomic indicators can not only be compared between different experimental groups, but also be comprehensively analyzed between different types of indicators. The last part of the formula It reflects the impact of the user allocation ratio of each layer on the atomic index. Here, Indicates The layer is assigned to The number of users in the experimental group, Indicates The total number of users in the layer, Indicates In this way, the formula can take into account the proportion of users at different levels, so that the atomic indicators can more accurately reflect the actual performance of the experimental group. The introduction of weights at each level makes it possible to fully consider the contribution of different user levels to the experimental results when calculating atomic indicators, thereby improving the scientificity and accuracy of the indicators. The calculation formula realizes the comprehensive consideration of user allocation probability, feature vector matching, atomic indicator standard value and user allocation ratio of each layer. This calculation method ensures the scientificity and accuracy of atomic indicators, so that the experimental results can fully reflect the performance of different experimental groups. In practical applications, the calculation of atomic indicators is of great significance for the analysis of the results of AB experiments. For example, in an AB test of an e-commerce website, if you need to compare the impact of two different page designs on user purchasing behavior, you can evaluate the performance of each experimental group by calculating atomic indicators such as click-through rate and conversion rate. By combining user allocation probability and feature vector matching, the impact of different page designs on user behavior can be more accurately quantified, thereby providing a scientific basis for page optimization and decision-making. In addition, the atomic indicator calculation method also has good adaptability and scalability. By adjusting the atomic indicator standard value and the weight of each layer, it can flexibly adapt to different types of AB experimental needs. For example, in an AB test of a news website, the effects of different experimental groups can be evaluated according to different atomic indicators (such as page dwell time, article sharing times, etc.), so as to have a more comprehensive understanding of users' reactions to different content.

[0101] Example 10: Based on the previous example, in step 3, the composite index is calculated using the following formula:

[0102] ;

[0103] in, The experimental group Composite index The value of is the standard value of the composite index, and different composite indexes correspond to different composite index standard values; Atomic Index The weight of ; is the number of atomic indices participating in the compound; It is Expected assignment to treatment group in stratum Number of users; is an atomic index The average value among all experimental groups.

[0104] Specific, composite indicators The calculation formula aims to integrate multiple atomic indicators , forming the experimental group Comprehensive performance evaluation. This process combines a variety of statistical techniques and mathematical methods to ensure that the composite indicator can fully and accurately reflect the performance of the experimental group. First, the first part of the formula represents the weighted sum of atomic indices. Here, is an atomic index The weight of These weights are used to adjust the importance of different atomic indicators in the composite indicator, ensuring that the composite indicator can comprehensively consider the influence of each atomic indicator. By adjusting the weights, it can be flexibly adapted to different experimental needs and goals. Next, the logarithmic transformation part of the formula The purpose is to smooth the cumulative effect of user allocation probabilities. The logarithmic transformation has the effect of compressing the data range so that the sum of user allocation probabilities does not excessively affect the value of the composite indicator in the calculation. This part ensures that even if the user allocation probability of some experimental groups is larger, its impact on the composite indicator will not be excessively amplified. The normalization part in the formula The deviation of the number of users in the experimental group in different layers is calculated. Here, Indicates The layer is assigned to The number of users in the experimental group, It is Expected assignment to treatment group in stratum By calculating the sum of squared deviations of the number of users and taking the square root, the formula normalizes the imbalance of user distribution at different levels, ensuring that the composite index is comparable between different experimental groups. In addition, the composite index also introduces a bias correction term , used to adjust the deviation of atomic indices. Here, is an atomic index The average value among all experimental groups. This part smoothes the differences between atomic indicators by calculating the deviation of each atomic indicator relative to its average value, ensuring that the composite indicator can more accurately reflect the comprehensive performance of the experimental group. The deviation correction term is introduced in the form of an exponential function, so that the experimental group with a large deviation of the atomic indicator is appropriately adjusted in the composite indicator, thereby avoiding the excessive influence of some atomic indicators on the composite indicator. Combining the above parts, the composite indicator The calculation formula of forms a comprehensive indicator that fully reflects the performance of the experimental group by integrating the weighted sum of atomic indicators, the logarithmic transformation of user allocation probability, the standardization of user number deviation, and the correction of atomic indicator deviation. This method ensures the scientificity and accuracy of the composite indicator, so that the experimental results can comprehensively and objectively evaluate the performance of different experimental groups. In practical applications, the calculation of composite indicators is of great significance for the analysis of the results of AB experiments. For example, in an AB test of an e-commerce website, the comprehensive impact of different page designs on user behavior can be evaluated by calculating multiple atomic indicators such as click-through rate and conversion rate and integrating them into composite indicators. By combining the weights and deviation correction of each atomic indicator, the comprehensive effect of different page designs on user behavior can be more accurately quantified, thereby providing a scientific basis for page optimization and decision-making. In addition, the composite indicator calculation method also has good adaptability and extensibility. By adjusting the weights and standard values ​​of atomic indicators, it can flexibly adapt to different types of AB experimental needs. For example, in an AB test of a news website, composite indicators can be calculated based on different atomic indicators (such as page dwell time, article sharing times, etc.), so as to have a more comprehensive understanding of users' reactions to different content.

[0105] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that these specific embodiments are only illustrative, and those skilled in the art may omit, replace, and change the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, merging the above method steps so as to perform substantially the same functions in substantially the same manner to achieve substantially the same results is within the scope of the present invention. Therefore, the scope of the present invention is limited only by the appended claims.

Claims

1. An AB experiment indicator management system based on user diversion, characterized in that: The system includes: a Flink real-time computing engine, which is used to obtain the behavior data of users in split tests in two or more variants; for each user, the behavior intensity, behavior participation depth, behavior value and behavior correlation are counted within a specified time window to form a feature vector; a user diversion part, which is used to reduce the dimension of the feature vector to obtain the reduced feature vector; based on the feature vector, a stratified sampling method is used to perform experimental grouping, and the grouping probability of each user is calculated to construct a full user table of all users; an experimental indicator calculation part, which is used to calculate the atomic indicator and composite indicator of each experimental group in the full user table; the feature vector is represented by the following formula: ； in, is a feature vector, which contains four dimensions, namely, behavior intensity , Depth of Behavioral Engagement , Behavioral Value and behavioral relevance ; is the total number of behavior types; For the The number of times the behavior occurs; For the The cumulative duration of the behavior, in seconds; For the The intensity index of the behavior; is the preset novelty attenuation coefficient; For the The novelty score of the behavior; A constant to prevent division by zero; For the preset User engagement score for each behavior; is the preset time attenuation coefficient; is the current timestamp; For the The timestamp of the last time the behavior occurred; For the The deep engagement score of each behavior; is the average depth of engagement across all behaviors; For the preset The value score of the behavior; For the preset The importance weight of the behavior; are the parameters of the Sigmoid function, which are used to adjust the steepness and center point of the curve; For the The relevance score of the behavior; For the The degree of completion of the behavior; Parameters that affect the frequency of the preset adjustment behavior; For the The frequency of the behavior; is the average frequency of all behaviors.

2. A method for managing AB experiment indicators based on user diversion, characterized in that: The method comprises: Step 1: Use the Flink real-time computing engine to obtain the behavior data of users in split tests in two or more variants. For each user, count their behavior intensity, behavior participation depth, behavior value, and behavior relevance within the specified time window to form a feature vector. Step 2: Reduce the dimension of the feature vector to obtain the reduced feature vector; perform experimental grouping using the stratified sampling method based on the reduced feature vector, calculate the grouping probability of each user, and construct a full user table for all users; Step 3: Calculate the atomic index and composite index of each experimental group in the full user table; In step 1, the feature vector is expressed using the following formula: ； in, is a feature vector, which contains four dimensions, namely, behavior intensity , Depth of Behavioral Engagement , Behavioral Value and behavioral relevance ; is the total number of behavior types; For the The number of times the behavior occurs; For the The cumulative duration of the behavior, in seconds; For the The intensity index of the behavior; is the preset novelty attenuation coefficient; For the The novelty score of the behavior; A constant to prevent division by zero; For the preset User engagement score for each behavior; is the preset time attenuation coefficient; is the current timestamp; For the The timestamp of the last time the behavior occurred; For the The deep engagement score of each behavior; is the average depth of engagement across all behaviors; For the preset The value score of the behavior; For the preset The importance weight of the behavior; are the parameters of the Sigmoid function, which are used to adjust the steepness and center point of the curve; For the The relevance score of the behavior; For the The degree of completion of the behavior; Parameters that affect the frequency of the preset adjustment behavior; For the The frequency of the behavior; is the average frequency of all behaviors.

3. The AB experiment indicator management method based on user diversion according to claim 2 is characterized in that: No. The intensity index of the behavior Calculated by the following formula: ； in, For the The average duration of the behavior; is the average duration of all behaviors; is the standard deviation of the duration of all behaviors; is the average number of steps of the 𝑖th behavior; is the average number of steps for all behaviors; is the standard deviation of the number of steps; No. Novelty score of the behavior Calculated using the following formula: ； in, is the frequency of occurrence of the 𝑖th behavior among all users; is the frequency value corresponding to the behavior with the highest frequency among all behaviors.

4. The AB experiment indicator management method based on user diversion according to claim 3 is characterized in that: No. The deep engagement score of the behavior Calculated using the following formula: ； in, is the total number of steps of the 𝑖th behavior; is the user's first behavior The time spent on the step; is the user's first behavior The number of operations on the step; The degree of completion of this behavior is equal to The number of steps completed for a behavior divided by the The total number of steps of the behavior; The relevance score of the behavior It is equal to the number of other behaviors that occur immediately after the 𝑖th behavior divided by the total number of behaviors.

5. The AB experiment indicator management method based on user diversion according to claim 4 is characterized in that: In step 2, the following formula is used to reduce the dimension of the feature vector to obtain the reduced dimension feature vector of: ； in, represents the t-distributed random neighbor embedding function; represents the uniform manifold approximation and projection function; Represents element-wise multiplication; is the feature vector The mean of is the feature vector The standard deviation of is a random noise term, which follows a normal distribution ; is a Gaussian kernel function that maps the feature vector to a high-dimensional space; is the Gaussian kernel matrix corresponding to the Gaussian kernel function; This is the transpose operation of a vector.

6. The AB experiment indicator management method based on user diversion according to claim 5, characterized in that: Step 3 specifically includes: according to the feature vector after dimensionality reduction Divide users into several layers and use the K-means clustering algorithm to achieve stratification: ； in, It is clusters; It is The centers of the clusters; for each layer, calculate its weight: ； in It is The number of users in the layer; is the subscript index; experimental groups; for each layer, users are allocated according to the preset ratio of the experimental groups; For the The preset ratio of the experimental groups is Layer assigned to The number of users in the group is: ； in, For the Layer assigned to The number of users in the group; is the rounding function.

7. The AB experiment indicator management method based on user diversion according to claim 6 is characterized in that: In step 2, the grouping probability of each user is calculated by the following formula: Users in Layer , which is assigned to The probability of a group is: ； in, Indicates experimental groups; Indicates user ; Is a user The reduced eigenvector of It is The center vector of each experimental group; is a preset adjustable smoothing parameter; a probability vector is constructed for each user through the following formula: ； The full user table is expressed using the following formula: ； in is the total number of users; For the The probability vector of each user; For the The reduced dimension feature vector of each user; Indicates The layer to which the user belongs.

8. The AB experiment indicator management method based on user diversion according to claim 7 is characterized in that: In step 3, the atomic index is calculated using the following formula: ； in, It is Atomic index of experimental group The value of Indicates The layer is assigned to The number of users in the experimental group; For atomic index The standard value of atomic index, different atomic indexes correspond to different standard values ​​of atomic index.

9. The AB experiment indicator management method based on user diversion according to claim 8, characterized in that: In step 3, the composite index is calculated using the following formula: ； in, The experimental group Composite index The value of is the standard value of the composite index, and different composite indexes correspond to different composite index standard values; Atomic Index The weight of ; is the number of atomic indices participating in the compound; It is Expected assignment to treatment group in stratum Number of users; is an atomic index The average value among all experimental groups.

Images