Experiment processing method and device, electronic equipment and computer readable medium

Abstract

The invention discloses an experiment processing method and device, electronic equipment and a computer readable medium, and relates to the technical field of computers.The specific implementation mode comprises the steps that an experiment processing request is received, an experiment project service identifier, an experiment state identifier and experiment resource data are obtained, and experiment processing is carried out according to the experiment project service identifier and the experiment resource data; determining an experiment value index and a total experiment sample number; in the current experiment period, according to the experiment state identification and the experiment value index, determining shunting rationality characteristics; determining a shunting method based on the experiment state identifier, and shunting the experiment total samples corresponding to the experiment total sample number based on the shunting method and the shunting rationality characteristics to obtain an experiment group and a control group; calculating a mean value difference between the experimental group and the control group, and further constructing a statistical magnitude based on the mean value difference; and constructing a rejection threshold, and if the statistical magnitude falls within the rejection threshold, rejecting the original hypothesis to obtain an experimental processing result. Therefore, the experiment processing accuracy and efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an experiment processing method, device, electronic device and computer-readable medium. Background Art

[0002] At present, when supply chain companies use AB experimental technology to achieve refined optimization of "efficiency", "cost" and "experience", they often encounter problems unique to the supply chain industry, namely, the long experimental cycle and the incomparability of the two groups before the experiment, which leads to poor accuracy and low efficiency of related experimental processing. Summary of the invention

[0003] In view of this, the embodiments of the present application provide an experimental processing method, device, electronic device and computer-readable medium, which can solve the problems of poor accuracy and low efficiency of existing experimental processing.

[0004] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present application, an experiment processing method is provided, including: receiving an experiment processing request, obtaining an experiment project business identifier, an experiment status identifier and experiment resource data, and determining an experiment value index and a total number of experiment samples according to the experiment project business identifier and the experiment resource data; in the current experiment cycle, determining a diversion rationality feature according to the experiment status identifier and the experiment value index; determining a diversion method based on the experiment status identifier, and diverting the total number of experiment samples corresponding to the total number of experiment samples based on the diversion method and the diversion rationality feature to obtain an experimental group and a control group; calculating the mean difference between the experimental group and the control group, and then constructing a statistic based on the mean difference; constructing a rejection threshold, and if the statistic falls within the rejection threshold, rejecting the null hypothesis to obtain an experimental processing result.

[0005] Optionally, determining the experimental value index and the total number of experimental samples includes: determining the experimental object based on the business identification of the experimental project, and then determining the experimental value index based on the experimental object; determining the corresponding experimental cost and experimental budget based on the business identification of the experimental project, and determining the number of samples that can be provided based on the experimental resource data; determining the total number of experimental samples based on the experimental cost, experimental budget and the number of samples that can be provided.

[0006] Optionally, determining diversion rationality features includes: if the experiment status identifier corresponds to the experiment not yet started, determining the number of experimental value indicators; if the number of experimental value indicators is greater than a preset threshold, performing linear regression on each experimental value indicator to obtain the corresponding key features; calculating the union of each key feature to obtain the diversion rationality feature.

[0007] Optionally, determining diversion rationality characteristics includes: if the experimental state identifier corresponds to that the experiment has currently started, converting the experimental value indicator into a discrete indicator through a causal forest model, and determining diversion rationality characteristics based on the discrete indicator.

[0008] Optionally, determining diversion rationality features includes: if the experimental status identifier corresponds to that the experiment has currently started, inputting the experimental value indicator into the causal random forest model to output the importance ranking and feature importance of the experimental value indicator; and determining diversion rationality features based on the importance ranking and feature importance.

[0009] Optionally, constructing a statistic based on the mean difference includes: determining a target parameter and a standard error based on the mean difference; and determining a ratio of the target parameter to the standard error as the statistic.

[0010] Optionally, constructing a rejection threshold includes: obtaining the first type of error probability, hybrid algorithm parameters and the number of times the experimental effect is observed in the experiment selected by the user; calling an offline table to determine the rejection threshold based on the first type of error probability, hybrid algorithm parameters and the number of times the effect is observed in the experiment.

[0011] In addition, the present application also provides an experimental processing device, including: a receiving unit, configured to receive an experimental processing request, obtain an experimental project business identifier, an experimental status identifier and experimental resource data, and determine an experimental value index and a total number of experimental samples based on the experimental project business identifier and the experimental resource data; a feature determination unit, configured to determine a diversion rationality feature in the current experimental cycle based on the experimental status identifier and the experimental value indicator; a diversion unit, configured to determine a diversion method based on the experimental status identifier, and divert the total experimental samples corresponding to the total number of experimental samples based on the diversion method and the diversion rationality feature to obtain an experimental group and a control group; a statistic construction unit, configured to calculate the mean difference between the experimental group and the control group, and then construct a statistic based on the mean difference; an experimental processing result determination unit, configured to construct a rejection threshold. If the statistic falls within the rejection threshold, the null hypothesis is rejected to obtain the experimental processing result.

[0012] Optionally, the receiving unit is further configured to: determine the experimental object according to the business identifier of the experimental project, and then determine the experimental value indicator according to the experimental object; determine the corresponding experimental cost and experimental budget according to the business identifier of the experimental project, and determine the number of samples that can be provided according to the experimental resource data; determine the total number of experimental samples according to the experimental cost, experimental budget and the number of samples that can be provided.

[0013] Optionally, the feature determination unit is further configured to: if the experiment status identifier corresponds to the experiment not yet started, determine the number of experiment value indicators; if the number of experiment value indicators is greater than a preset threshold, perform linear regression on each experiment value indicator to obtain the corresponding key feature; calculate the union of each key feature to obtain the diversion rationality feature.

[0014] Optionally, the feature determination unit is further configured to: if the experiment status identifier corresponds to that the experiment has currently started, convert the experiment value indicator into a discrete indicator through a causal forest model, and determine the diversion rationality feature based on the discrete indicator.

[0015] Optionally, the feature determination unit is further configured to: if the experimental status identifier corresponds to that the experiment has currently started, input the experimental value indicator into the causal random forest model to output the importance ranking and feature importance of the experimental value indicator; and determine the diversion rationality feature based on the importance ranking and feature importance.

[0016] Optionally, the statistic construction unit is further configured to: determine a target parameter and a standard error based on the mean difference; and determine a ratio of the target parameter to the standard error as a statistic.

[0017] Optionally, the experimental processing result determination unit is further configured to: obtain the first type of error probability, hybrid algorithm parameters and the number of times the experimental effect is observed in the experiment selected by the user; call the offline table to determine the rejection threshold based on the first type of error probability, hybrid algorithm parameters and the number of times the effect is observed in the experiment.

[0018] In addition, the present application also provides an experimental processing electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the experimental processing method as described above.

[0019] In addition, the present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned experimental processing method is implemented.

[0020] To achieve the above objective, according to another aspect of the embodiments of the present application, a computer program product is provided.

[0021] A computer program product according to an embodiment of the present application includes a computer program, and when the program is executed by a processor, the experimental processing method provided by the embodiment of the present application is implemented.

[0022] One embodiment of the above invention has the following advantages or beneficial effects: the present application receives an experimental processing request, obtains an experimental project business identifier, an experimental status identifier and experimental resource data, determines the experimental value index and the total number of experimental samples according to the experimental project business identifier and the experimental resource data; determines the rationality characteristics of diversion according to the experimental status identifier and the experimental value index in the current experimental cycle; determines the diversion method based on the experimental status identifier, diverts the total number of experimental samples corresponding to the total number of experimental samples based on the diversion method and the rationality characteristics of diversion to obtain an experimental group and a control group; calculates the mean difference between the experimental group and the control group, and then constructs a statistic based on the mean difference; constructs a rejection threshold, and if the statistic falls within the rejection threshold, rejects the original hypothesis to obtain the experimental processing result. Thus, the accuracy and efficiency of experimental processing can be improved.

[0023] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application.

[0025] Figure 1 is a schematic diagram of the main process of an experimental processing method according to an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of the main process of an experimental processing method according to an embodiment of the present application;

[0027] Figure 3 It is a schematic diagram of the main flow of an experimental processing method according to an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of the main units of the experimental processing device according to an embodiment of the present application;

[0029] Figure 5 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0030] Figure 6 It is a structural diagram of a computer system of a terminal device or server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following is an explanation of the exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of known functions and structures is omitted in the following description. It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of the present application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of the present application, certain software, components, models, and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or must have used the solution. In the technical solution of the present application, the collection, collection, update, analysis, processing, use, transmission, storage, etc. of the user personal information involved are in compliance with the provisions of relevant laws and regulations, are used for legal and reasonable purposes, and do not violate public order and good customs, are not shared, leaked or sold outside these legal uses, and are subject to supervision and management by regulatory authorities. Take necessary measures to prevent illegal access to such user personal information data, maintain the security of user personal information, network security and national security, ensure that persons who have access to personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once such user personal information data is no longer needed, the risk should be minimized by limiting or even prohibiting data collection and / or deleting data.

[0032] When used, including in certain related applications, user privacy is protected by de-identifying data, such as by removing specific identifiers when used, controlling the amount or specificity of data stored, controlling how data is stored, and / or other methods of de-identification.

[0033] Figure 1 is a schematic diagram of the main process of the experimental processing method according to an embodiment of the present application, such as Figure 1 As shown, the experimental processing method mainly includes the following steps S101 to S105.

[0034] Step S101, receiving an experiment processing request, obtaining an experiment project business identifier, an experiment status identifier and experiment resource data, and determining an experiment value index and a total number of experiment samples according to the experiment project business identifier and the experiment resource data.

[0035] In this embodiment, the execution subject of the experimental processing method (for example, it can be a server) can receive the experimental processing request through a wired connection or a wireless connection. The experimental processing can be to collect data step by step and perform real-time analysis when there is an intervention in the case that there are differences between the experimental group and the control group before the experiment, so as to make a quick decision. When the execution subject receives the experimental processing request, it can obtain the experimental project business identifier, experimental status identifier and experimental resource data carried in the request. Among them, the experimental project business identifier can be used to characterize the experimental application scenario, and the experimental cost and the configured experimental budget in the experimental application scenario can be determined according to the experimental project business identifier.

[0036] Specifically, determining the experimental value index and the total number of experimental samples includes: determining the experimental object according to the business identification of the experimental project, and then determining the experimental value index according to the experimental object; determining the corresponding experimental cost and experimental budget according to the business identification of the experimental project, and determining the number of samples that can be provided according to the experimental resource data; determining the total number of experimental samples according to the experimental cost, experimental budget and the number of samples that can be provided.

[0037] For example, the experimental objects can be sales departments, deliverymen, etc., and the experimental value indicators can be C-acquisition timeliness rate, successful delivery timeliness rate, customer complaints, etc. Determine the experimental cost and the associated experimental budget based on the business identifier of the experimental project. The experimental resource data may include deliverymen's job categories, successful delivery orders, successful delivery timeliness rate, C-acquisition timeliness rate, after-sales interval orders, B-acquisition orders, C-acquisition orders, etc. According to the data that needs to be used as samples, the number of data that can be used as samples is determined from the experimental resource data as the number of samples that can be provided. Select all or part of the available samples to participate in the experiment based on the experimental cost and experimental budget, so as to determine the number of samples participating in the experiment as the total number of experimental samples.

[0038] Step S102, in the current experimental cycle, determine the diversion rationality characteristics according to the experimental status identifier and the experimental value index.

[0039] The execution entity may divide the experiment into multiple experimental cycles. Specifically, the number of times K to be analyzed during the experiment may be set, and each experimental cycle may be obtained according to the determined total duration T.

[0040] Specifically, determining the rationality characteristics of the diversion includes: if the experimental status identifier corresponds to that the experiment has not yet started, determining the number of experimental value indicators; if the number of experimental value indicators is greater than a preset threshold, performing linear regression on each experimental value indicator to obtain the corresponding key characteristics; calculating the union of each key feature to obtain the rationality characteristics of the diversion.

[0041] If the experiment has not yet started, you can perform offline regression on as many features and value indicators as possible that have been collected. After that, count all features with p-values ​​less than 0.05 after linear regression as key features. If there are multiple value indicators, such as C-acquisition timeliness rate + successful delivery timeliness rate, perform linear regression on the two indicators respectively, count the two groups of key features, and find the union of the two groups of key features, and finally use them as the total key feature, that is, the diversion rationality feature.

[0042] Step S103, based on the experimental state identifier, determine the diversion method, and divert the total number of experimental samples corresponding to the total number of experimental samples based on the diversion method and diversion rationality characteristics to obtain an experimental group and a control group.

[0043] If the experimental status identifier corresponds to that the experiment has not started, the diversion method is determined to be random hash diversion. If the experimental status identifier corresponds to that the experiment has started, the diversion method is determined to be diversion using the PSM method. Based on the obtained diversion rationality characteristics, the determined diversion method is executed on the total experimental samples to obtain the experimental group and the control group.

[0044] Step S104, calculating the mean difference between the experimental group and the control group, and then constructing a statistic based on the mean difference.

[0045] The parameters and standard errors are estimated based on the calculated mean differences, and statistics are constructed based on the estimated parameters and standard errors to be used for comparison with the rejection threshold to obtain the experimental treatment results.

[0046] Step S105, construct a rejection threshold. If the statistic falls within the rejection threshold, reject the null hypothesis and obtain the experimental processing result.

[0047] The rejection threshold refers to the range of values ​​of the statistic for rejecting the null hypothesis in hypothesis testing. It is an area determined based on the distribution of the test statistic and the given significance level. If the specific value of the test statistic calculated using the sample observations falls within the rejection region, the null hypothesis is rejected; otherwise, the null hypothesis cannot be rejected.

[0048] Specifically, constructing a rejection threshold includes: obtaining the first-class error probability selected by the user (e.g., α=0.05), the hybrid algorithm parameters (e.g., = 0.5) and the number of times the experimental effect is observed in the experiment (e.g., 5 times); call the offline table to determine the rejection threshold (e.g., C) based on the first type of error probability, the hybrid algorithm parameters, and the number of times the effect is observed in the experiment. i =2.4135).

[0049] This embodiment receives an experimental processing request, obtains the experimental project business identifier, the experimental status identifier and the experimental resource data, determines the experimental value index and the total number of experimental samples according to the experimental project business identifier and the experimental resource data; determines the diversion rationality characteristics according to the experimental status identifier and the experimental value index in the current experimental cycle; determines the diversion method based on the experimental status identifier, diverts the total experimental samples corresponding to the total number of experimental samples based on the diversion method and the diversion rationality characteristics to obtain the experimental group and the control group; calculates the mean difference between the experimental group and the control group, and then constructs a statistic based on the mean difference; constructs a rejection threshold, and if the statistic falls within the rejection threshold, rejects the original hypothesis to obtain the experimental processing result. This can improve the accuracy and efficiency of experimental processing.

[0050] Figure 2 is a schematic diagram of the main flow of an experimental processing method according to an embodiment of the present application, such as Figure 2 As shown, the experimental processing method mainly includes the following steps S201 to S207.

[0051] Step S201, receiving an experiment processing request, obtaining an experiment project business identifier, an experiment status identifier and experiment resource data, and determining an experiment value index and a total number of experiment samples according to the experiment project business identifier and the experiment resource data.

[0052] Experimental processing can be to collect data step by step and conduct real-time analysis when there is an intervention in order to make a quick decision when there is a difference between the experimental group and the control group before the experiment. When the execution subject receives the experimental processing request, it can obtain the experimental project business identifier, experimental status identifier and experimental resource data carried in the request. Among them, the experimental project business identifier can be used to characterize the experimental application scenario, and the experimental cost and the configured experimental budget under the experimental application scenario can be determined according to the experimental project business identifier.

[0053] Step S202, in the current experimental cycle, determine the diversion rationality characteristics according to the experimental status identifier and the experimental value index.

[0054] Specifically, determining the rationality characteristics of the diversion includes: if the experimental state identifier corresponds to that the experiment has currently started, converting the experimental value indicator into a discrete indicator through a causal forest model, and determining the rationality characteristics of the diversion based on the discrete indicator.

[0055] For example, if the experiment has already started, you can consider using more complex causal inference methods, such as causal forest models, causal random forest models, etc. In the causal forest model, the value index needs to be converted into a discrete index. For example, a guy's C-grab timeliness rate is 0.90, and his historical average C-grab timeliness rate is 0.88, and the expected C-grab timeliness rate increase is exactly 2%. Then, because 0.9>0.88*(1.02), it can be considered that the guy's experiment is successful, and thus marked as 1, otherwise marked as 0. In this way, the continuous C-grab timeliness rate variable is converted into a discrete variable of type 0, 1, and then the discrete variable of type 1 can be selected as the rationality feature of the diversion.

[0056] Specifically, determining the rationality features of the diversion includes: if the experimental status identifier corresponds to that the experiment has currently started, inputting the experimental value indicator into the causal random forest model to output the importance ranking and feature importance of the experimental value indicator; and determining the rationality features of the diversion based on the importance ranking and feature importance.

[0057] For example, the causal random forest model can be directly trained. After the model is trained, the diversion rationality features are input into the causal random forest model to output the importance ranking and feature importance of each feature, and the top-ranked and more important features are selected as diversion rationality features.

[0058] Step S203, based on the experimental state identifier, determine the diversion method, and divert the total number of experimental samples corresponding to the total number of experimental samples based on the diversion method and diversion rationality characteristics to obtain an experimental group and a control group.

[0059] Step S204, calculating the mean difference between the experimental group and the control group.

[0060] Step S205, determining target parameters and standard errors based on the mean differences.

[0061] Step S206: determine the ratio of the target parameter to the standard error as a statistic.

[0062] For example, if the current time is the first cycle of the experiment, that is, the time point is t=K, all N*K samples obtained at this time are input into the DID experiment, and the t statistic T of the regression coefficient of DID is calculated at this time. 1 , the corresponding p value is P 1 Similarly, at each t=K,2K…,[T / K]K, the mean difference between the two groups (i.e., the experimental group and the control group) is estimated based on the collected sample information. Here, the mean difference is denoted as d, and the target parameter estimated based on the current mean difference is denoted as , and the standard error is recorded as , then we can construct the statistic T i = / At this time, the construction parameter C 1 , C 2 , …, C S , and under the original hypothesis that the experimental effect is not significant, so that:

[0063]

[0064] Even if the rejection threshold is constructed in this way, once the observed , then under the original hypothesis. For any i, , that is, the observation falls into the rejection threshold. At this time, the original hypothesis is considered to be invalid, that is, the hypothesis that there is no significant difference between the experimental group and the control group is not valid. This means that there is a significant difference between the experimental group and the control group after the experiment, and it is believed that the experiment has worked at this time.

[0065] Step S207, construct a rejection threshold. If the statistic falls within the rejection threshold, reject the null hypothesis and obtain the experimental processing result.

[0066] To construct a specific C i The rejection threshold can be constructed by using the Pocock boundary method and O'Brien-Fleming boundary method introduced in the GST early stopping algorithm, or by searching the preset offline table. When the first type of error probability α is set to 0.05, if the parameter Set to 0.5, and want to observe the experimental effect 5 times in the experiment, then the rejection threshold C at this time i You can choose 2.4135.

[0067] For example, in actual experiments, the statistics T of the early stopping DID hybrid algorithm are continuously counted i , the statistic T i With the rejection threshold C i (At this time C i can be equal to 2.4135) for comparison. Once T i >2.4135, that is, T i >C i , that is, the statistic falls within the rejection threshold, then the experiment can be stopped and the null hypothesis (the null hypothesis is that the experimental effect is not significant) can be rejected. The experimental treatment result obtained is that the experimental effect is significant. This scheme will be continuously tested over time, so that decisions can be made in the early stages of data accumulation. This efficiency not only saves resources, but also shortens decision-making time and improves the flexibility and practicality of the testing process.

[0068] Figure 3 FIG. 1 is a schematic diagram of the main flow of an experimental processing method according to an embodiment of the present application. Figure 3 As shown, the experimental processing method of the embodiment of the present application can first read the AB experiment configuration, and then determine the number of early stopping statistics of the experiment, divert the samples according to the sample historical data to obtain the experimental group and the control group, enter the DID, and then count the early stopping DID algorithm statistics for the current experiment, and judge whether the early stopping DID algorithm statistics meet the conditions. If not, wait for the next time to read new data. If so, end the experiment.

[0069] Among them, DID refers to the DID double difference method: it can solve the situation where there are differences between the two groups before the experiment, but it cannot perform stepwise analysis experiments. It is usually used in observational studies or quasi-experimental designs, and is particularly suitable for longitudinal data with treatment groups and control groups. Its basic principle is to estimate the causal effect of the intervention by comparing the changes in the treatment group and the control group before and after the intervention. DID does not have the ability to stop the experiment early: DID requires the specification of a period of experimental time, and it cannot be used for hypothesis testing in the process of stepwise data accumulation. The solution of this application, by designing a new statistic, can perform hypothesis testing in the process of stepwise data accumulation. DID is not suitable for sequential design: in terms of experimental design, DID is usually a quasi-experimental design or a natural experiment. It is not suitable for sequential design. The solution of this application can perform real-time data analysis and calculation by gradually collecting data.

[0070] Early stopping refers to the sequential probability ratio test (SPRT): the experiment can be stopped early, but it cannot handle the situation where the two groups are not comparable before the experiment. It is mainly used for hypothesis testing in the process of gradual data accumulation. The basic principle of SPRT is to calculate the likelihood ratio of the observed data and compare it with the pre-set threshold to decide whether to accept or reject the hypothesis, or continue to collect data. The early stopping algorithm currently cannot handle the problem that the experimental group and the control group are not comparable before the experiment: it cannot handle the scenario where the experimental group and the control group are not equal before the experiment, but only have parallel trends. However, the early stopping DID algorithm can be compatible with the problem that the experimental group and the control group are not comparable before the experiment by eliminating this difference (that is, the experimental group and the control group are not equal before the experiment). Through creative statistical design, the advantages of both techniques are achieved while avoiding their disadvantages.

[0071] The experimental processing method of the embodiment of the present application can be divided into three stages in total:

[0072] Determine the experiment configuration and the number of early stop analysis: First, determine the experiment object, duration, etc. according to the specific experiment project business and experiment resources, and then divide the reasonable number of experiment analysis;

[0073] Experimental sample diversion: Select reasonable experimental sample characteristics, decide on the diversion method to be used based on whether the experimental group is specified, and test the diversion results to obtain a reasonable grouping plan;

[0074] Early stopping DID hybrid algorithm statistics calculation: After obtaining a reasonable grouping method in the previous stage, enter the DID experiment, calculate the early stopping DID hybrid algorithm, and determine whether early stopping is needed; if early stopping is not possible, wait for the next sample update.

[0075] Phase 1: Determine the experimental configuration and the number of early stopping analyses: The purpose of this phase is to clarify the requirements through specific business scenarios, so as to determine the configuration of various aspects of the experiment.

[0076] 1. Determine the experimental value indicator: The experimental value indicator is the purpose of the experiment. Taking the sales department as an example, such indicators usually include C-acquisition timeliness rate, successful delivery timeliness rate, customer complaints, etc. These indicators are related to cost, efficiency, and experience. Determining the value indicator is to determine the purpose of the experiment, and thus determine a method to quantify the experimental results. This indicator can be determined based on historical business standards.

[0077] 2. Determine the experimental object: The experimental object is the direct object related to the value indicator in the first step. For example, for the C-acquisition timeliness rate, the direct object is the sales department, the salesperson, etc. However, it may be found that the experiment needs to be in one dimension, while the statistics of the value indicator may be in another dimension. For example, when the experimental operation has an impact on the salespeople of the entire sales department, and the salespeople of a sales department influence each other on the value indicator of this experiment, although the experimental entity is still the salesperson, the statistics of the value indicator need to be upgraded to the C-acquisition timeliness rate of the sales department dimension to obtain a more stable and robust experimental result.

[0078] 3. Determine the duration of the experiment and the number of experimental analyses: In this part, you need to consider how much the experiment costs and what is the acceptable range. For example, in the case of project establishment, more experimental samples will be more convincing. For example, it is preferred to use 10,000 guys in the sales department to do the experiment instead of 2,000 guys. However, due to cost constraints, it may not be possible to expand the experimental results to 10,000 guys to do the experiment. On the other hand, the business hopes that the experiment can be done for 3 months, but due to cost and budget, as well as various requirements, there may only be 1 month of experimental time. Therefore, it was finally determined that the time was 1 month to conduct the experiment on 2,000 guys. When the total duration of the experiment and the total number of samples are determined, the next step is to determine the number of experimental analyses, which is also the key configuration for early stopping. For example, there are N guys now, and the total duration is determined to be T, then it is necessary to determine an integer K (that is, the number of times K to be analyzed during the experiment, which is an additional parameter determined by project experience). This K divides the time T into T / K parts. At each t=K, 2K…, [T / K]K, the early stopping DID hybrid algorithm statistics are calculated for the experiment. If the analysis results of the statistics at this time show that the experiment can be stopped (that is, it falls within the rejection threshold), then stop the experiment in advance and reject the null hypothesis. Otherwise, wait for the next data update and continue to analyze the experimental results.

[0079] Phase 2: Experimental sample diversion phase: After determining the basic configuration and information of the experiment, in this phase, all samples need to be divided into reasonable experimental and control groups to ensure that there is no difference between the two groups in terms of various value indicators and key features.

[0080] 1. Determine the rationality of the diversion features used by the sample: Let's take the C-acquisition timeliness experiment of the sales department brother as an example. At this time, the value indicator of the experiment is the C-acquisition timeliness, and the object of the experiment is the brother. Then we need to find all the features that are relatively related to the C-acquisition timeliness as the rationality indicators of diversion. Commonly used methods here include linear regression and more complex causal inference methods, etc.

[0081] Among them, linear regression: If the experiment has not yet started, you can collect as many features and value indicators as possible for offline regression. After that, count all the features with p-values ​​less than 0.05 after linear regression as key features. If there are multiple value indicators, such as C-timely rate + timely delivery rate, perform linear regression on the two indicators respectively, count the two sets of key features, and find the union of the two sets of key features, and finally use them as the total key features. For example, now there is a candidate feature pool X 1 , X 2 , X 3 , X 4 After regression of the C-class on-time rate, we find It is a key indicator. After regression of the timely delivery rate, it is found that is the key feature. Then the final rationality indicator feature is their union .

[0082] Among them, causal inference: If the experiment has already started at this time, then you can consider using more complex causal inference methods, such as causal forest model, causal random forest model, etc. In the causal forest model, the value index needs to be converted into a discrete index. For example, the C-grab timeliness rate of a certain guy is 0.90, and his historical average C-grab timeliness rate is 0.88, and the expected C-grab timeliness rate increase is just 2%. Then because 0.9>0.88*(1.02), it can be considered that the experiment of this guy is successful, so it is marked as 1, otherwise it is marked as 0, so the continuous C-grab timeliness rate variable is converted into a discrete variable of type 0, 1. In addition, other discretization methods can also be considered for conversion. However, the causal random forest does not require this conversion, and the model can be trained directly. After the model is trained, the importance ranking and feature importance of each feature are output, and the top-ranked and more important features are selected as the rationality features of the diversion.

[0083] 2. Conduct diversion experiments on experimental samples.

[0084] If the experiment has not started yet, use random hashing to split: If the experiment has not started yet, you can use the random hashing method to split all experimental samples. After the split is completed, use t-test, TOST equivalent analysis and other methods to test the rationality of the split. If the test passes, you can start the experiment.

[0085] If the experiment has already begun, use propensity score matching (PSM) and other methods to divert: If the experiment has already begun, then the samples that have already begun the experiment will be used as the natural experimental group. Then, this part of the samples can be used as the experimental group and marked as 1, and the remaining samples can be used as the control group and marked as 0, and then PSM matching can be used to obtain appropriate experimental group and control group samples.

[0086] Phase 3: Early Stop DID Hybrid Algorithm Statistics Calculation

[0087] Based on the above stages, we obtained the required value indicators and rationality test indicators, and divided the experimental group and the control group. We also determined the experimental duration T and the number of analyses K during the experiment. Next, we entered the early stopping DID hybrid algorithm.

[0088] First, suppose there are N experimental objects. Within the experimental time T, N*T experimental data can be generated. The experiments are analyzed at the time points t=K, 2K…, [T / K]K respectively. Here, [T / K]=s can be used to describe the key calculations of the early stopping DID hybrid algorithm starting from one cycle.

[0089] If the current time is the first cycle of the experiment, that is, the time point is t=K, all the N*K samples obtained at this time are input into the DID experiment, and the t statistic T of the regression coefficient of DID is calculated at this time. 1 , the corresponding p value is P 1 Similarly, at each t=K,2K…,[T / K]K, the mean difference between the two groups is estimated based on the collected sample information. Here, the mean difference is recorded as d, and the parameter estimated based on the current mean difference is recorded as , and the standard error is recorded as , then we can construct the statistic T i = / At this time, the construction parameter C 1 , C 2 , …, C S , and under the original hypothesis that the experimental effect is not significant, so that:

[0090]

[0091] Even if the rejection threshold is constructed in this way, once the observed , then under the original hypothesis. For any i, , that is, the observation falls into the rejection threshold. At this time, the original hypothesis is considered to be invalid, that is, the hypothesis that there is no significant difference between the experimental group and the control group is not valid. This means that there is a significant difference between the experimental group and the control group after the experiment, and it is believed that the experiment has worked at this time.

[0092] The remaining question is the parameters here The construction of can use the typical exponential function form:

[0093]

[0094] To construct a specific C i You can use the Pocock boundary method and O'Brien-Fleming boundary method introduced in the GST early stopping algorithm to construct the rejection threshold, or search the offline table, such as:

[0095] What this means is that when the probability of the first type of error α is set to 0.05, if the parameter Set to 0.5, and want to observe the experimental effect 5 times in the experiment, then the constant (i.e. rejection threshold) C i You can choose 2.4135. In this way, the core calculation structure of the entire early stopping DID hybrid algorithm is described.

[0096] For example, the experimental resource data can come from the project to improve the timely delivery rate of the salesmen in each sales department in a certain region. Since we don't know whether there is a difference in the timely delivery rate between the experimental group and the control group before the experiment, we can't use the traditional early stop experiment. So we use the above method to calculate the experimental effect using the early stop DID hybrid algorithm.

[0097] The experimental resource data includes:

[0098] Job category: This refers to the category or identity of the courier or delivery person, which may be used to distinguish different types of work tasks or responsibilities. For example, some couriers may specialize in delivery, while others may be responsible for picking up packages;

[0099] Delivery: This refers to the number of packages successfully delivered to customers. Delivery means that the package has been successfully delivered and signed for by the customer;

[0100] Timely delivery rate: This refers to the proportion of parcels successfully delivered within the specified time. The calculation method is usually the number of parcels delivered on time divided by the total number of parcels delivered, reflecting the efficiency and punctuality of delivery;

[0101] C-end collection timeliness rate: This refers to the proportion of C-end collection tasks that are successfully completed within the specified time. C-end collection tasks refer to the collection of parcels of a specific category, such as after-sales returns, collection of specific customer groups, etc. The calculation method is similar to the timely delivery rate, which is the number of C-end collection tasks completed in time divided by the total number of C-end collection tasks;

[0102] After-sales interval order volume: This refers to the number of packages involved in the after-sales service process, which may include the number of packages processed after-sales such as returns and exchanges;

[0103] B pick-up volume: This refers to the number of parcels in B-type pick-up tasks. B-type pick-up tasks refer to corporate-type pick-up tasks, and the specific definition may vary depending on the company's internal regulations;

[0104] C collection volume: This refers to the number of parcels collected by the C-end. The definition of C collection tasks may also vary depending on the company's internal regulations, and usually refers to parcel collection tasks for specific customer groups or specific types of parcels.

[0105] For example, in the experimental resource data, idnex_no is used to represent the brother's id, group_name is used to represent the group identifier, A is used to represent the experimental group identifier, and B is used to represent the control group identifier. For example, in the experimental resource data, the brother with id 31724 is divided into the experimental group. At this time, there is a difference in the timely delivery rate between the experimental group and the control group. The experimental group is about 0.9808, and the control group is about 0.9758. The difference before the experiment is 0.518%, but in this experiment, the user hopes to have a 0.5% improvement after the experiment. However, before the experiment, the two groups already had a difference of about 0.5%, which makes the two groups incomparable before the experiment, and it is difficult to explain the improvement of the experimental effect after the experiment, so the traditional early stopping method cannot be used to calculate the experimental effect. Therefore, continue to use the early stopping DID hybrid algorithm proposed in this application for calculation. At this time, the first type of error probability is selected as 0.05, and the parameter Set it to 0.5, and if you want to observe the experimental results 5 times in the experiment, then according to the offline table, the constant c can be selected as 2.4135. In the actual experiment, the statistics of the early stopping DID hybrid algorithm T are continuously counted. i , T i and C i For comparison, once T i >2.4135, that is, T i >C i , that is, the statistic falls within the rejection threshold, then the experiment can be stopped and the null hypothesis can be rejected.

[0106] As can be seen from the above, this scheme is compatible with the differences between the first two groups of experiments, and at the same time, it can perform hypothesis testing in the process of gradual data accumulation. Its reliability is mainly reflected in its solid theoretical foundation, effective error rate control, high efficiency and real-time decision-making ability:

[0107] 1. Theoretical basis: The theoretical basis of this scheme relies on calculating the statistics of the observed data and comparing it with a pre-set threshold. The threshold is set according to the required error rate.

[0108] 2. Error rate control: This solution can effectively control the first type of error (false positive rate). By setting appropriate thresholds, the required error rate level can be predetermined when designing the experiment. This design ensures that the actual error rate will not exceed the preset level in long-term use. This control capability makes this solution very reliable in applications, especially in scenarios where strict control of error rates is required.

[0109] 3. Efficiency and sample size: Compared with traditional fixed sample size testing methods, this scheme generally requires a smaller sample size to make a decision. This is because this scheme will continue to test over time, so that decisions can be made at an early stage of data accumulation. This efficiency not only saves resources, but also shortens the decision time and improves the flexibility and practicality of the testing process.

[0110] 4. Real-time decision-making: This solution can analyze and make decisions at any time as data is gradually accumulated. This real-time decision-making capability is very important in many practical applications. For example, in clinical trials, it can detect treatment effects or side effects early, and adjust the trial design or take corresponding measures in time. In industrial quality control, it can quickly detect abnormalities in the production process, and intervene in time to reduce losses.

[0111] This solution expands the capabilities of early stopping experiments through the new early stopping DID hybrid algorithm statistics: it can also conduct early stopping experiments when there are differences between the experimental group and the control group before the experiment, and can also perform real-time experimental analysis and decision-making. This solution expands the functions of the DID algorithm by introducing the idea of ​​step-by-step analysis of the early stopping algorithm: after introducing the core idea of ​​the early stopping algorithm, the DID algorithm can perform hypothesis testing and data analysis in real time, and is not limited to the original scenario where a certain experimental time must be specified.

[0112] Figure 4 Schematic diagram of the main units of the experimental processing device according to the embodiment of the present application. Figure 4 As shown, the experiment processing device 400 includes a receiving unit 401, a feature determination unit 402, a flow distribution unit 403, a statistics construction unit 404 and an experiment processing result determination unit 405.

[0113] The receiving unit 401 is configured to receive an experiment processing request, obtain an experiment project business identifier, an experiment status identifier and experiment resource data, and determine the experiment value index and the total number of experiment samples according to the experiment project business identifier and the experiment resource data.

[0114] The feature determination unit 402 is configured to determine the diversion rationality feature according to the experiment status identifier and the experiment value index in the current experiment cycle.

[0115] The diversion unit 403 is configured to determine a diversion method based on the experimental state identifier, and divert the total experimental samples corresponding to the total number of experimental samples based on the diversion method and diversion rationality characteristics to obtain an experimental group and a control group.

[0116] The statistics constructing unit 404 is configured to calculate the mean difference between the experimental group and the control group, and then construct a statistic based on the mean difference.

[0117] The experimental processing result determination unit 405 is configured to construct a rejection threshold. If the statistic falls within the rejection threshold, the null hypothesis is rejected and the experimental processing result is obtained.

[0118] In some embodiments, the receiving unit 401 is further configured to: determine the experimental object according to the business identifier of the experimental project, and then determine the experimental value indicator according to the experimental object; determine the corresponding experimental cost and experimental budget according to the business identifier of the experimental project, and determine the number of samples that can be provided according to the experimental resource data; determine the total number of experimental samples according to the experimental cost, experimental budget and the number of samples that can be provided.

[0119] In some embodiments, the feature determination unit 402 is further configured to: if the experiment status identifier corresponds to the experiment not yet started, determine the number of experiment value indicators; if the number of experiment value indicators is greater than a preset threshold, perform linear regression on each experiment value indicator to obtain the corresponding key feature; calculate the union of each key feature to obtain the diversion rationality feature.

[0120] In some embodiments, the feature determination unit 402 is further configured to: if the experiment status identifier corresponds to that the experiment has currently started, convert the experiment value indicator into a discrete indicator through a causal forest model, and determine the diversion rationality feature based on the discrete indicator.

[0121] In some embodiments, the feature determination unit 402 is further configured to: if the experimental status identifier corresponds to that the experiment has currently started, input the experimental value indicator into the causal random forest model to output the importance ranking and feature importance of the experimental value indicator; and determine the diversion rationality feature based on the importance ranking and feature importance.

[0122] In some embodiments, the statistics construction unit 404 is further configured to: determine a target parameter and a standard error based on the mean difference; and determine a ratio of the target parameter to the standard error as a statistic.

[0123] In some embodiments, the experimental processing result determination unit 405 is further configured to: obtain the first type of error probability, hybrid algorithm parameters and the number of times the experimental effect is observed in the experiment selected by the user; call the offline table to determine the rejection threshold based on the first type of error probability, hybrid algorithm parameters and the number of times the effect is observed in the experiment.

[0124] It should be noted that the experimental processing method and the experimental processing device of the present application have a corresponding relationship in terms of specific implementation content, so the repeated content will not be explained again.

[0125] Figure 5 An exemplary system architecture 500 to which the experiment processing method or the experiment processing device according to the embodiment of the present application can be applied is shown.

[0126] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0127] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0128] The terminal devices 501 , 502 , and 503 may be various electronic devices having an experiment processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0129] Server 505 may be a server that provides various services, such as a background management server that provides support for experimental processing requests submitted by users using terminal devices 501, 502, and 503 (only as an example). The background management server may receive an experimental processing request, obtain an experimental project business identifier, an experimental status identifier, and experimental resource data, and determine the experimental value index and the total number of experimental samples according to the experimental project business identifier and the experimental resource data; determine the rationality characteristics of diversion according to the experimental status identifier and the experimental value index in the current experimental cycle; determine the diversion method based on the experimental status identifier, and divert the total number of experimental samples corresponding to the total number of experimental samples based on the diversion method and the rationality characteristics of diversion to obtain an experimental group and a control group; calculate the mean difference between the experimental group and the control group, and then construct a statistic based on the mean difference; construct a rejection threshold, and if the statistic falls within the rejection threshold, reject the original hypothesis to obtain the experimental processing result. This can improve the accuracy and efficiency of experimental processing.

[0130] It should be noted that the experiment processing method provided in the embodiment of the present application is generally executed by the server 505 , and accordingly, the experiment processing device is generally set in the server 505 .

[0131] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0132] Reference below Figure 6, which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present application. Figure 6 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0133] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0134] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0135] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present application are executed.

[0136] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0137] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The units involved in the embodiments described in the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a receiving unit, a feature determination unit, a diversion unit, a statistics construction unit, and an experimental processing result determination unit. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0139] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device receives an experimental processing request, obtains the experimental project business identifier, the experimental state identifier and the experimental resource data, and determines the experimental value index and the total number of experimental samples according to the experimental project business identifier and the experimental resource data; in the current experimental cycle, according to the experimental state identifier and the experimental value index, determines the rationality characteristics of the diversion; based on the experimental state identifier, determines the diversion method, and diverts the total number of experimental samples corresponding to the total number of experimental samples based on the diversion method and the rationality characteristics of the diversion to obtain the experimental group and the control group; calculates the mean difference between the experimental group and the control group, and then constructs a statistic based on the mean difference; constructs a rejection threshold, and if the statistic falls within the rejection threshold, rejects the original hypothesis and obtains the experimental processing result.

[0140] The computer program product of the present application includes a computer program, and the computer program implements the experimental processing method in the embodiment of the present application when executed by a processor.

[0141] According to the technical solution of the embodiment of the present application, the accuracy and efficiency of experimental processing can be improved.

[0142] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. An experimental processing method, characterized in that: include: Receive an experiment processing request, obtain an experiment project business identifier, an experiment status identifier, and experiment resource data, and determine an experiment value indicator and a total number of experiment samples based on the experiment project business identifier and the experiment resource data; In the current experimental cycle, determining the rationality characteristics of the diversion according to the experimental status identifier and the experimental value indicator; Based on the experimental state identifier, determine a diversion method, and divert the total number of experimental samples corresponding to the total number of experimental samples based on the diversion method and the diversion rationality characteristics to obtain an experimental group and a control group; Calculating the mean difference between the experimental group and the control group, and then constructing a statistic based on the mean difference; A rejection threshold is constructed. If the statistic falls within the rejection threshold, the null hypothesis is rejected and the experimental processing result is obtained.

2. The method according to claim 1, characterized in that The determination of the experimental value index and the total number of experimental samples includes: Determine the experimental object according to the business identifier of the experimental project, and then determine the experimental value indicator according to the experimental object; Determine the corresponding experimental cost and experimental budget according to the experimental project business identifier, and determine the number of samples that can be provided according to the experimental resource data; The total number of experimental samples is determined according to the experimental cost, the experimental budget and the available number of samples.

3. The method according to claim 1, characterized in that The determining of the rationality characteristics of diversion includes: If the experiment status indicator corresponds to that the experiment has not yet started, determining the number of the experiment value indicator; If the number of the experimental value indicators is greater than a preset threshold, linear regression is performed on each of the experimental value indicators to obtain corresponding key features; The union of each of the key features is calculated to obtain the diversion rationality feature.

4. The method according to claim 1, characterized in that The determining of the rationality characteristics of diversion includes: If the experimental status identifier corresponds to that the experiment has currently started, the experimental value indicator is converted into a discrete indicator through a causal forest model, and the rationality feature of the diversion is determined based on the discrete indicator.

5. The method according to claim 1, characterized in that The determining of the rationality characteristics of diversion includes: If the experiment status identifier corresponds to that the experiment has currently started, input the experiment value indicator into the causal random forest model to output the importance ranking and feature importance of the experiment value indicator; According to the importance ranking and the feature importance, the diversion rationality feature is determined.

6. The method according to claim 1, characterized in that The constructing a statistic based on the mean difference includes: determining a target parameter and a standard error based on the mean difference; The ratio of the target parameter and the standard error is determined as a statistic.

7. The method according to any one of claims 1 to 6, characterized in that The construction rejection threshold includes: Obtain the first-class error probability, hybrid algorithm parameters, and the number of times the experimental effect is observed in the experiment selected by the user; The offline table is called to determine a rejection threshold based on the first type of error probability, the hybrid algorithm parameters and the number of times the effect is observed to be achieved in the experiment.

8. An experimental processing device, characterized in that: include: A receiving unit is configured to receive an experiment processing request, obtain an experiment project business identifier, an experiment status identifier and experiment resource data, and determine an experiment value indicator and a total number of experiment samples according to the experiment project business identifier and the experiment resource data; A feature determination unit is configured to determine a diversion rationality feature according to the experiment status identifier and the experiment value indicator in a current experiment cycle; A diversion unit is configured to determine a diversion method based on the experimental state identifier, and divert the total number of experimental samples corresponding to the total number of experimental samples based on the diversion method and the diversion rationality feature to obtain an experimental group and a control group; A statistics construction unit, configured to calculate the mean difference between the experimental group and the control group, and then construct a statistic based on the mean difference; The experimental processing result determination unit is configured to construct a rejection threshold. If the statistic falls within the rejection threshold, the original hypothesis is rejected to obtain the experimental processing result.

9. An electronic device for experimental processing, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images