Small sample AB experiment shunt analysis system combining double-sample verification and differential calculation

Abstract

The invention relates to the technical field of internet software testing, and discloses a small sample AB experiment shunt analysis system combining double-sample verification and differential calculation. The system comprises a shunting decision generation module, a parallel computing and analysis module, an evaluation and learning module and a knowledge migration module. The method comprises the following steps: synchronously generating a main shunting decision and a verification shunting decision with short-term balance for each user request; effect difference values based on the two decisions are calculated in parallel; evaluating a decision confidence coefficient through a calibration formula fusing sample size information; performing asymmetric weight modeling of attenuation priority according to the confidence so as to update the shunting model; and knowledge migration of a new experiment scene is realized by using the dynamic feature association network. According to the method, the difference consistency feedback closed loop is constructed, so that the AB experiment shunting strategy can be self-iteratively optimized based on historical evidences, and the initialization balance, the conclusion reliability and the system learning efficiency of the experiment are improved under the small sample condition.

Description

Technical Field

[0001] This invention relates to the field of Internet software testing technology, and more specifically, to a small-sample A / B test triage analysis system that combines two-sample verification and difference calculation. Background Technology

[0002] In the rapid iteration and refined operation of internet products, A / B testing is a key method for evaluating the effectiveness of strategies. By diverting user traffic to different strategy versions and comparing their core metrics, quantitative evidence can be provided for decision-making. However, when conducting feature rollouts, rapid trial and error, or testing with niche user groups, the constraint of limited user sample size is often encountered. In small-sample scenarios, ensuring the statistical reliability of experimental conclusions and accelerating the identification of valid signals are crucial to the efficiency of iteration.

[0003] Existing A / B testing splitting techniques generally employ simple random algorithms or hashing methods based on fixed rules for user grouping. While these methods are stable with large sample sizes, they have significant limitations in small-sample scenarios. First, due to the inherent volatility of randomness, simple random splitting at the initial stage of an experiment can easily lead to accidental imbalances in the distribution of key user characteristics between the experimental and control groups. This baseline difference can contaminate subsequent evaluations of the strategy's true effectiveness. Second, existing splitting systems typically use static or predefined splitting models, lacking the ability to self-adjust based on historical experimental performance. Each experiment is treated as an independent event, preventing the system from learning from past validation results to optimize the initial splitting quality for future experiments, making it difficult to systematically improve experimental efficiency. Furthermore, when faced with entirely new experimental configurations or those with sparse feature combinations, the system lacks an effective knowledge transfer mechanism, often requiring a longer "cold start" period to accumulate data, further prolonging the time to obtain reliable conclusions.

[0004] Therefore, the core issues that current technology needs to address are: how to improve the intelligence and balance of initial triage in A / B experiments, especially in small sample scenarios; how to establish a mechanism that enables the triage system to iterate and optimize itself based on historical experimental feedback; and how to accelerate the system's learning process when facing new experimental scenarios, so as to obtain effective experimental conclusions more quickly and reliably under limited sample constraints. Summary of the Invention

[0005] This invention provides a small-sample A / B test triage analysis system that combines two-sample validation and differential computation. This system addresses the baseline imbalance problem caused by the initial randomness of triage in small-sample A / B tests, as well as the challenge of slow statistical conclusion establishment. This system constructs a differential consistency feedback framework, which proactively optimizes the triage strategy and continuously iterates using historical experimental data, thereby improving experimental efficiency and the reliability of conclusions.

[0006] The core architecture of the system includes a traffic splitting decision generation module, a parallel computing and analysis module, an evaluation and learning module, and a knowledge transfer module. The traffic splitting decision generation module synchronously generates a main traffic splitting decision and a verification traffic splitting decision for each user request; the parallel computing and analysis module independently groups and calculates user behavior data based on these two decisions, obtaining the main difference scores respectively. Difference from benchmark The evaluation and learning module calculates decision confidence and updates model parameters through differential consistency evaluation units and weight calculators; the knowledge transfer module uses dynamic feature association networks to achieve knowledge transfer across experiments. All modules work together to collect, analyze, evaluate, and update strategies.

[0007] Furthermore, the constrained random benchmark generator ensures that the validation split sequences remain proportionally stable within any consecutive decision window by maintaining a decision buffer queue and imposing short-term statistical equilibrium constraints. This mechanism provides a low-variance and unbiased control benchmark for effect evaluation, effectively suppressing the interference of random fluctuations on experimental conclusions.

[0008] Furthermore, the differential consistency evaluation unit adopts the formula Calculate the decision confidence level CD, where the dynamic stability compensation is... Combined sample size With historical fluctuations Information. This design enables confidence assessments to have inherent sample size calibration capabilities, automatically outputting conservative assessments when the data base is weak, thereby providing a reliability guarantee for model learning.

[0009] Furthermore, the weight calculus executes an asymmetric weight evolution strategy, the rule of which mandates that when the decision confidence CD falls below a low threshold... When CD is not lower than a certain value, the weight decay operation is performed first; only when CD is not lower than a certain value will the weight decay operation be performed first. Furthermore, directional gain adjustment is only performed when both high confidence and significant effect conditions are met simultaneously. This "attenuation-first" mechanism prevents low-quality signals from misleading the model and ensures the stability of the learning process.

[0010] Furthermore, the adaptive exploration mechanism integrated into the dynamic triage model dynamically adjusts the exploration coefficients based on the learning history of feature patterns and introduces corresponding random perturbations into the triage decision. This design proactively explores features with insufficient experience or unstable conclusions, while fully utilizing mature and stable features.

[0011] Furthermore, the system constructs a dynamic feature association network that encodes the relationships between features using a graph structure and dynamically evolves based on experimental confidence. Based on this network, the system generates migration compensation weights for new experiments by retrieving neighboring features and fusing their historical weights. This method enables cross-scenario knowledge transfer and alleviates the cold start problem.

[0012] Furthermore, the parallel computing and analysis module employs a staged incremental computing approach, continuously outputting local difference value sequences. This not only supports the final evaluation but also provides high-frequency data for monitoring experimental trends, enabling the system to perceive dynamic changes in experimental results.

[0013] Furthermore, the differential consistency assessment unit fine-tunes the decision confidence level by analyzing the trend and volatility of the local difference sequence. This calibration based on process signals enhances the system's ability to judge interim experimental results.

[0014] Furthermore, the three-layer cascaded feature encoding method adopted by the system integrates experimental configuration, contextual information, and time-phase data, providing a structured and information-complete feature representation foundation for model learning and knowledge transfer.

[0015] Through the above technical solutions, the present invention system realizes the evolution of the traffic allocation strategy from static random assignment to dynamic continuous optimization. This system is particularly suitable for small sample scenarios, comprehensively improving the initialization quality, learning efficiency, and adaptability to new scenarios of A / B testing by ensuring traffic balance, intelligently evaluating experimental signals, robustly updating model parameters, and efficiently transferring historical knowledge.

[0016] The technical effects and advantages of this invention are as follows: Compared to existing methods that rely on simple random or static hashing for stream allocation, this invention introduces a constrained random benchmark generator to produce validation stream allocation decisions independent of the main model. This generator, through internal state caching and proportional constraints, ensures that its output sequence satisfies strict statistical equilibrium within a short-term window. The system computes two sets of effect differences based on the main and validation decisions in parallel, and calculates the decision confidence of the deviation between the two through a calibrated evaluation unit that integrates current sample size and historical fluctuation information. This mechanism provides a dynamic, stable, and unbiased reference benchmark for evaluating the effect of each experiment, enabling the system to effectively distinguish between genuine policy effects and spurious signals caused by accidental stream allocation imbalances or small sample noise. This provides a purer and more reliable feedback source for the optimization learning of the main stream allocation model, thereby reducing the risk of misjudgment due to random fluctuations in the early stages of the experiment.

[0017] Compared to existing technologies that treat each experiment as an isolated event and maintain a fixed diversion strategy, this invention designs a weight calculus that executes an asymmetric weight evolution strategy and assigns it a higher priority for conservative decay adjustment. This calculus selectively enhances or decays the parameters in the dynamic diversion model based on the confidence evaluation results and effect direction of each experiment. The model also integrates an adaptive exploration factor negatively correlated with feature learning maturity and historical evaluation stability. This series of designs enables the main diversion model to dynamically adjust its diversion tendency based on continuous historical evidence and proactively increase randomness to explore potential better solutions when the signal is ambiguous. The resulting effect is that the diversion strategy possesses the ability to continuously self-improve, and the sensitivity and efficiency of experiments can cumulatively improve with system runtime. Simultaneously, the priority mechanism ensures the robustness of the learning process when facing low-quality data, preventing the continuous amplification of error patterns in the closed loop from an algorithmic perspective.

[0018] Compared to existing technologies that lack cross-experimental knowledge transfer capabilities and suffer from a cold-start process from scratch when facing new scenarios, this invention constructs and maintains a feature association network that dynamically adjusts weights based on feature co-occurrence relationships and experimental results, and maintains representative weight vectors at the feature unit level for this network. When processing new experimental requests with sparse features, the system retrieves semantically similar historical feature neighbors through this network and fuses their representative weights to generate a transfer compensation term, which participates in the initial weight setting of the new experiment. This approach allows new experiments to draw on the scattered experience of similar past experiments, rather than starting completely randomly. Its implementation mechanism achieves structured storage and scenario-based transfer of historical knowledge, resulting in a significant reduction in the sample accumulation and trial-and-error cycle required for new or niche experiments to obtain stable and reliable distribution strategies, and improving the overall adaptability and operational efficiency of the system when handling diverse and long-tail experimental needs. Attached Figure Description

[0019] Figure 1 This is the overall flowchart of the small-sample AB experiment diversion analysis system of the present invention.

[0020] Figure 2 This is a flowchart of the dual-path diversion decision generation process of the present invention.

[0021] Figure 3 This is a flowchart of the decision confidence assessment and weight evolution of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1 As attached Figures 1 to 3 The system shown is a small-sample A / B experiment triage analysis system that combines two-sample validation and difference calculation. The system achieves continuous self-improvement of the triage strategy by constructing a closed loop of data acquisition, evaluation learning and decision optimization.

[0024] The core workflow of the system is as follows: User requests are simultaneously entered into the dynamic triage model and the constrained random benchmark generator, generating dual-path triage labels; user behavior data is processed in parallel according to the dual labels to calculate the difference value; after the experiment, the evaluation learning module calculates the decision confidence and updates the model parameters and feature association network accordingly; the updated model takes effect immediately and applies to subsequent triage decisions, thus forming a closed loop.

[0025] The system is configured with a series of global parameters, and their typical example values ​​and functions are as follows: Basic Diversion Ratio It is set to 0.5, used to determine the basis for traffic splitting; Benchmark decision queue length The threshold for balance deviation is 100. The value is 0.05, and both together control the short-term balance of the verification sequence; Low decision confidence threshold The threshold for high decision confidence is 0.3. The effect threshold is 0.7. It is 0.01, used for conditional judgment in weight calculation; Weight decay rate The weight learning rate is 0.1. The value is 0.05, which controls the strength of conservative attenuation and directional gain, respectively; Stability compensation coefficient The value is 1.96, used to adjust the conservatism of the confidence level calculation; Number of trend analysis windows The threshold for trend significance is 10. The volatility threshold is 0.001. The confidence level is 0.005, indicating an upward adjustment. and reduction amount All are 0.05; these parameters are used for trend calibration. Explore coefficient parameters and The values ​​are 0.05 and 0.03, respectively, used to adjust the balance between exploration and exploitation; feature association enhancement rate. The attenuation rate is 0.1. It is set to 0.01 to control the evolution of the feature network; Number of neighbors for knowledge transfer The value is 5, which determines the number of historical features to be referenced during migration.

[0026] S100: Experiment Initialization and Cold Start The system receives a command to create a new A / B experiment. The command includes an experiment identifier, strategy descriptions for the experimental and control groups, and the key performance indicators to be observed. The system allocates resources for the experiment and initializes them.

[0027] In the constrained random benchmark generator, an empty decision cache queue is created for this experiment identifier. In the weight store upon which the dynamic splitting model relies, initial weight values ​​are pre-set for the feature patterns that the experiment may involve. During the initial system deployment or cold start phase where historical data is completely missing, the weight parameters for all experiments are... Initialized to zero, dynamic feature association network Empty.

[0028] At this stage, due to all weights The base split probability output by the dynamic split model is zero. The value remains constant at 0.5. The adaptive exploration mechanism is affected by the number of updates. And has a high exploration coefficient This results in a high degree of randomness in the initial stages of the diversion decision.

[0029] The system will initially behave as a standard stochastic splitter with short-term equilibrium constraints. After the initial experiments are completed, the system will begin generating non-zero decision confidence (CD) values ​​and updating weights and feature association networks through subsequent learning steps. It also begins to build, thus entering a state of transition from random exploration to learning-based intelligent triage.

[0030] S200: Dual-path diversion decision generation When a user request carrying a user identifier, experiment identifier, and context information arrives at the system entry point, the traffic splitting decision generation module is triggered, and two independent traffic splitting decision processes are executed simultaneously.

[0031] S201: Main Flow Diversion Decision Generation The dynamic traffic splitting model processes this request. First, it performs feature encoding on the request's context information to generate a feature vector. .

[0032] The feature vector Generates through a three-layer cascaded encoding: the first layer performs one-hot encoding on the experiment type and core indicators to obtain a vector. The second layer performs multi-hot encoding on user attribute tags and request context information to obtain a vector. The third layer encodes the experimental time phases, which are divided according to the percentage of users who have cumulatively entered the system since the start of the experiment relative to the total planned sample size. Each 10% represents a phase, and the system generates an encoding vector based on the phase to which the current request belongs. .final, ,in This indicates a vector concatenation operation.

[0033] The model is based on Use the key to query its parameter matrix. A two-dimensional bias weight vector is obtained through linear transformation. ,in This is an optional feature nonlinear mapping function. The basic splitting probability is calculated as follows: , This is the sigmoid function.

[0034] Furthermore, the model invokes an adaptive exploration mechanism, which maintains an update counter for each feature pattern. And a recording loop of recent decision confidence (CD). Exploration coefficient. Calculated dynamically according to the following formula: ; in The coefficient of variation is the recent CD record. The model generates uniform random perturbations. Calculate the final diversion probability , For the amplitude limiting function, , The system is based on Perform Bernoulli trials to generate the main branching decision. (0 represents the control group, and 1 represents the experimental group).

[0035] S202: Verify the generation of the diversion decision A constrained random benchmark generator processes the request synchronously. The generator receives the user identifier and experiment identifier, generates a deterministic hash value using a hash function (such as MurmurHash3), and maps it to... Pseudo-random numbers within the interval .

[0036] Calculate the initial decision , This is an indicator function. The generator queries the decision cache queue corresponding to this experiment, whose length is... If the current length of the queue is less than Then output directly. And add it to the queue.

[0037] If the queue is full, the simulation will... After adding to the queue, calculate the proportion of experimental group decisions in the queue. .like Then output ; Otherwise, to satisfy the short-run equilibrium constraint, the output... The actual output Elements are pushed into the queue, and the first element is removed, thus updating the queue state. This mechanism ensures that any consecutive elements are pushed into the queue. In each validation decision, the proportion of the experimental group and... The absolute deviation does not exceed .

[0038] Furthermore, the verification of the triage decision The generation process is completely independent of the internal state and weight parameters of the dynamic splitting model. This ensures that it provides an unbiased and statistically stable random benchmark for evaluating the effectiveness of the main diversion decision.

[0039] S300: User Behavior Data Collection and Parallel Streaming Aggregation User-generated behavioral events are asynchronously collected and sent to a message queue. Jobs implemented using a stream processing framework (such as Apache Flink) continuously consume this event stream.

[0040] S301: Dual-path aggregation state maintenance This task maintains two independent sets of keyed aggregate states for each experiment identifier. The first set of states is named (experiment ID, ...). The second set of states uses (experiment ID, ...) as the key, and (experiment ID, ...) as the key. The key is ). Each state set accumulates the target metric values ​​(such as click count) and unique user counts under its key. These two state sets are physically isolated to ensure that data is counted independently based on the primary and secondary grouping methods.

[0041] S302: Staged Window Calculation and Sequence Generation Streaming job configuration uses a rolling window based on processing time (e.g., window length). The calculation is triggered at the end of each window.

[0042] For each experiment, the job retrieves the newly added cumulative value within the current window period from both sets of aggregate states, and calculates the index rate based on the main split within that window. and the rate of performance based on verification-based traffic splitting Then, the local principal difference values ​​of this window are calculated. and local benchmark difference These window-level difference values, along with their timestamps, are written to the time-series database in real time. The aggregated state itself continues to accumulate and is not cleared between windows; it is used to calculate the global value at the end of the experiment.

[0043] S400: Experiment Termination and Global Difference Calculation When the experiment reaches the preset termination conditions (such as the total duration reaching 24 hours or the total sample size), Once the threshold of 10000 is reached, the system marks it as finished. The stream processing job outputs the cumulative data for the entire experiment cycle. Based on the cumulative data from the main stream splitting decision, the global main difference value is calculated. Based on the cumulative data from validating the triage decision, the global baseline difference score is calculated. At the same time, record the total effective sample size of the experiment. .

[0044] S500: Decision Confidence Assessment and Calibration The experiment end event triggers the differential consistency evaluation unit in the evaluation and learning module.

[0045] S501: Core Confidence Calculation Differential Consistency Evaluation Unit Receives , and Based on the type and core indicators of this experiment, the unit retrieves historical experiments of similar types from the historical experiment database. Value. If the number of historical experiments is sufficient (e.g., greater than or equal to 30), then calculate its sample standard deviation. Otherwise, a preset global conservative estimate is used as... Subsequently, the unit calculates the dynamic stability compensation amount. Finally, calculate the decision confidence level (CD): ; The CD value range is Its size reflects and The relative significance of the difference.

[0046] As a preferred embodiment, the dynamic stability compensation amount It is similar in form to the estimation of the standard error of the sample mean. Among them, The effect size was estimated ( The overall fluctuation of ) This represents the improvement in estimation accuracy due to the current experimental sample size. (Constant) The value of corresponds to the critical value of the confidence level commonly used in statistics, providing an interpretable scale for the compensation amount.

[0047] Introduction Primarily serving as a minimal positive term to prevent division by zero errors, it ensures that the formula accurately reflects the effect size. and Computational robustness when all values ​​are close to zero. Sample size. The direct impact on the reliability of the conclusions is mainly controlled through the confidence threshold in the subsequent weight calculation strategy.

[0048] S502: Confidence Calibration Based on Early Trends Furthermore, the differential consistency evaluation unit queries the time series database for the most recent experiment. A series of windows sequence .

[0049] The unit performs linear regression on the sequence to fit the trend slope. and the standard deviation of the residuals .

[0050] like and If the experimental effect shows a clear and stable trend, the confidence level should be increased. .

[0051] Otherwise, if the trend is unclear or there is significant noise, the confidence level should be lowered. . The final decision confidence level after calibration will be used in subsequent weight calculation steps.

[0052] S600: Asymmetric Weight Evolution and Model Update The weighted calculator receives the feature vector. , and Perform the update.

[0053] S601: Weight Calculation Strategy Execution The weight calculator calculates the weights based on the eigenvectors. The query retrieves the corresponding propensity weight vector. The calculus executes a prioritized asymmetric evolution strategy: 1. Conservative attenuation adjustment: If Then calculate the effective attenuation coefficient. And the weights are decayed: This operation shrinks the weights towards zero, aiming to reduce the impact of low-confidence experiments. After execution, the process jumps to S603.

[0054] 2. Directional gain adjustment: Only when Only when this condition is met will this step be performed. If both conditions are met simultaneously... and Then the gain is triggered.

[0055] First, determine the direction: If Let the direction vector be... ;like ,set up Calculate the gain. Update weights: .

[0056] The calculator will be updated Persist to weight storage and update the parameter matrix in reverse. Moderation and characteristics The relevant parts.

[0057] S602: Synchronous Evolution of Feature Association Networks As a preferred implementation, the system constructs and maintains a dynamic feature association network, which learns synchronously based on each experiment. This network is a weighted undirected graph. ,node Basic feature unit, edge weight This represents the correlation strength. The feature set used in this experiment is denoted as . (Depend on (Analysis yielded).

[0058] The evolution rule of the network is as follows: for each distinct pair of feature units i and j in S, update the weights of their edges to... .for Middle feature unit and not present Feature units in The edges between them are updated with their weights. .

[0059] The edge weights are constrained to be non-negative, which means that when At that time, it will be placed directly. This design ensures the sparsity of the feature association network, effectively severing the associations between feature units that do not co-occur for a long time or repeatedly do not co-occur in low-confidence experiments, thereby improving network retrieval efficiency and the targeting of knowledge transfer.

[0060] Furthermore, the system is for each basic feature unit Maintain a "representative weight vector" Maintaining this "representative weight vector" involves accumulating the weight information scattered across different experiments and feature combinations into the basic feature unit level, providing a queryable data foundation for knowledge transfer when processing new feature combinations later.

[0061] Whenever a feature vector weight When updated, for Each feature unit contained Its representative weight is based on Perform a smooth update. Fusion rate. It is a small positive number (e.g., 0.01) that controls how quickly new weight information is incorporated into historical representative weights. Small The value makes It reflects more long-term, average historical experience, is more robust to noise, and constructs stable and reliable feature-level knowledge priors.

[0062] S603: Hot Deployment of Models The configuration management service detects changes in weight storage and pushes the updated model parameters (or weight vector indexes) to all service instances deploying the dynamic traffic splitting model in real time. Each instance completes a hot reload of the parameters in memory within milliseconds, after which all new user requests will be split based on the optimized model.

[0063] S700: Knowledge Transfer for New Experimental Scenarios When the system processes a feature sparse (e.g., feature vector) The knowledge transfer module is activated when the proportion of zero elements in China and Africa is less than 30% or when it is a completely new experiment.

[0064] S701: Neighbor Feature Retrieval and Weight Fusion Knowledge Transfer Module Analysis The set of its active feature units is obtained. The module is used in dynamic feature association networks. In China, with Starting from the node, calculate the other nodes in the network. arrive Comprehensive correlation strength .

[0065] Select The highest value Each node forms a neighbor set. .like Empty or all If the sum is zero, then the migration compensation vector will be... Set it to the zero vector. Otherwise, the module retrieves it from the feature representative weight storage. Each node The corresponding representative weight vector The migration compensation vector is generated by weighted averaging: ; S702: Application of Transfer Weights When performing the S601 weight calculation step for the first time in this new experiment, the weight calculator will use Instead of using the default zero vector, the initial weight vector corresponding to the experimental feature is used for evolution calculation.

[0066] The steps S100 to S700 above constitute the complete process of the system processing a full A / B experiment and achieving self-iteration. Feedback signals are obtained through two-sample validation and differential calculation, the robustness of the evolutionary direction is ensured through confidence assessment and asymmetric learning, and knowledge is accumulated and transferred through feature association networks, thereby continuously improving the effectiveness and reliability of the diversion strategy under small sample conditions.

[0067] Finally, the following points should be noted: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection", and "linkage" should be interpreted broadly, and can be mechanical or electrical connections, or internal connections between two components, or direct connections. "Up", "down", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may change. Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other. In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A small-sample A / B test triage analysis system combining two-sample validation and difference calculation, characterized in that, The system is configured to perform the following functions: The routing decision generation module synchronously generates a main routing decision and a verification routing decision for each user request in an A / B experiment. The main routing decision is generated by the dynamic routing model based on the current model weights and request characteristics, while the verification routing decision is generated by a constrained random benchmark generator independent of the dynamic routing model. The parallel computing and analysis module groups user behavior data according to the main traffic splitting decision and the verification traffic splitting decision, and independently calculates the main difference value. Difference from benchmark ; The evaluation and learning module includes a differential consistency evaluation unit and a weight calculator. The differential consistency evaluation unit is used to evaluate the weights based on the... and The decision confidence level CD is calculated using the weight calculus based on the experimental feature vector. The above And the CD, executes an asymmetric weight evolution strategy to update the weight parameters of the dynamic splitting model; The knowledge transfer module, including a dynamic feature association network, is used to store the associations between feature units and to provide weight transfer compensation when dealing with new experiments with sparse features. The dynamic traffic splitting model generates the main traffic splitting decision for subsequent user requests based on the updated weight parameters, thus forming a closed-loop system that continuously optimizes the traffic splitting strategy based on historical dual-sample verification feedback.

2. The small-sample AB experiment triage analysis system combining two-sample validation and difference calculation as described in claim 1, characterized in that, The constrained random benchmark generator is configured to maintain a length of [length missing] for each experiment. A decision cache queue; generates initial random decisions based on user identifiers and experiment identifiers; Determine whether adding the decision to the decision cache queue violates the following constraint: in any consecutive In this decision-making process, the proportion of decisions made by the experimental group was compared with the pre-set baseline diversion proportion. The absolute difference does not exceed the threshold. ,in , The initial random decision is set as a preset constant; if the constraint is violated, the opposite decision is output to ensure that the constraint is met; if the constraint is not violated, the initial random decision is output.

3. The small-sample AB experiment triage analysis system combining two-sample validation and difference calculation as described in claim 1, characterized in that, The formula for calculating the decision confidence level (CD) by the differential consistency assessment unit is as follows: ; in, The dynamic stability compensation amount is calculated using the following formula: , A constant greater than 0 Historical experiments categorized according to experiment type and core indicators Statistical standard deviation of the value This represents the total number of valid user samples within the current experimental period.

4. The small-sample AB experiment triage analysis system combining two-sample validation and difference calculation as described in claim 1, characterized in that, The asymmetric weight evolution strategy executes the following steps sequentially: (a) If Then, conservative attenuation adjustment is performed on the experimental feature vector. The associated weight parameters are scaled and decayed towards zero, with the decay coefficient being... Positive correlation; (b) If Then, further judgment is made: if both conditions are met simultaneously... and Then directional gain adjustment is performed along... The direction indicated by the symbol increases the weighting parameter, and the gain magnitude is related to... Positive correlation; in, and The confidence threshold. , The effect threshold is greater than 0, and the regulatory priority of step (a) is higher than that of step (b).

5. The small-sample AB experiment triage analysis system combining two-sample validation and difference calculation as described in claim 1, characterized in that, The dynamic triage model integrates an adaptive exploration mechanism; the adaptive exploration mechanism maintains an exploration coefficient for each feature pattern. The The calculation is negatively correlated with the historical number of times the weight parameters corresponding to the feature pattern have been updated, and positively correlated with the coefficient of variation of the CD value obtained from recent experiments containing that feature pattern; when making decisions, the model generates a value within the interval Uniformly distributed random perturbation And the base probability calculated based on weights and After summing and limiting, the final splitting probability is obtained.

6. The small-sample AB experiment triage analysis system combining two-sample validation and difference calculation as described in claim 1, characterized in that, The dynamic feature association network is a graph structure, where nodes are basic feature units and edge weights reflect the association strength between feature units. The network dynamically adjusts the weights of relevant edges based on the decision confidence (CD) of each experiment. The adjustment rule is as follows: for feature unit pairs that appear together in the current experiment, the weight of their edges is increased. For feature unit pairs that did not co-occur in this experiment, the weight of their edges is reduced. Or remain unchanged, where and For the learning rate, and .

7. The small-sample AB experiment triage analysis system combining two-sample validation and difference calculation as described in claim 6, characterized in that, When processing new experiments with sparse features, the knowledge transfer module is configured to perform the following operation: using the feature vector of the new experiment... The feature unit in the query is used as the starting point, and the system retrieves the element with the highest sum of edge weights in the dynamic feature association network. Each neighboring feature unit; by weighted average of the... The historical weight parameters corresponding to each neighbor feature unit are used to generate a migration compensation vector, and the weighting coefficients are the corresponding edge weights; the migration compensation vector is used as the initial weight parameters of the new experiment and input into the weight calculator.

8. The small-sample AB experiment triage analysis system combining two-sample validation and difference calculation according to claim 1, characterized in that, The parallel computing and analysis module adopts a phased incremental computing method, calculating and storing the principal difference scores at the window level according to a preset time window or sample size window. Difference from benchmark This forms a difference value sequence, which is used to monitor the trend and fluctuation of the experimental results.

9. The small-sample AB experiment triage analysis system combining two-sample validation and difference calculation according to claim 8, characterized in that, The differential consistency evaluation unit is further configured to evaluate the principal difference value. The window sequence is subjected to trend analysis, and the decision confidence level (CD) is calibrated; the trend analysis includes linear fitting of the window sequence to obtain the trend slope. The calibration includes: if the trend slope The absolute value exceeds the preset trend threshold. And the sequence fluctuation is lower than the preset fluctuation threshold. If the CD value is positive, then the CD value is increased; otherwise, the CD value is decreased.

10. The small-sample AB experiment triage analysis system combining two-sample validation and difference calculation according to claim 1, characterized in that, The experimental feature vector Generated through a three-layer cascaded encoding: the first layer vector uses one-hot encoding to encode the experiment type and core indicators; The second-layer vector uses multi-hot encoding to encode user attributes and request context; the third-layer vector uses scalar encoding of stage numbers or one-hot encoding to encode experimental time stages; the feature vector F is formed by concatenating the first, second, and third-layer vectors.

Images