Method, apparatus, medium, and device for testing information publishing strategy interaction
By designing multiple controlled experiments and using resampling methods to calculate the mean and variance of pre-set test statistics, the problem of difficulty in measuring the interaction of multiple information dissemination strategies in existing technologies is solved, and more efficient strategy evaluation and optimization are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TENCENT TECHNOLOGY (SHENZHEN) CO LTD
- Filing Date
- 2021-03-04
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies struggle to accurately measure the interaction between multiple information dissemination strategies in A/B testing, especially when multiple strategies are in effect simultaneously. Traditional methods such as ANOVA and multivariate testing suffer from assumption limitations and computational complexity.
By designing multiple controlled experiments, the mean and variance of the pre-set test statistics are calculated using the resampling method. The interaction between information release strategies is judged based on the confidence interval. A hierarchical diversion mechanism is adopted to optimize user group segmentation and reduce computation and storage difficulty.
Accurately evaluating and optimizing information dissemination strategies can more precisely measure the interaction between multiple strategies, reduce computational and storage complexity, and improve verification efficiency.
Smart Images
Figure CN115017028B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of information dissemination, specifically to a method, apparatus, medium, and equipment for testing the interaction of information dissemination strategies. Background Technology
[0002] Generally, when conducting A / B testing (also known as group isolation experiments), only the individual effect of the experimental strategy is usually considered. However, on an experimental platform, numerous experimental strategies may be effective simultaneously. These strategies can influence each other.
[0003] Experimental platforms often use Analysis of Variance (ANOVA) to measure the interactions between experimental strategies. However, ANOVA requires three assumptions: independence, normality, and homogeneity of variance, limiting its application. Some platforms employ Multivariate Testing (MVT), but variance estimation methods in MVT require storing large amounts of data, making them difficult to implement in engineering and potentially inaccurate. Summary of the Invention
[0004] To accurately verify the interaction between multiple strategies, this application provides a method, apparatus, medium, and device for testing the interaction of information dissemination strategies. The technical solution is as follows:
[0005] Firstly, this application provides a method for testing the interaction of information dissemination strategies, the method comprising:
[0006] Based on multiple preset information dissemination strategies, a control experiment is determined for each of the information dissemination strategies; the experimental variable of the control experiment is whether or not the information dissemination strategy is implemented to users in the target user group.
[0007] The control experiment was conducted on the target user group to obtain the experimental result data corresponding to the target user group.
[0008] Based on the experimental results data, determine the mean of the preset test statistic;
[0009] Based on the resampling method, the variance of the preset test statistic is calculated from the experimental results data.
[0010] Based on the mean and the variance, determine the confidence interval of the preset test statistic;
[0011] Based on the confidence interval of the preset test statistic, the test results of the interaction of the multiple information release strategies are determined.
[0012] Secondly, this application provides an apparatus for verifying the interaction of information dissemination strategies, the apparatus comprising:
[0013] The experiment design module is used to determine a control experiment corresponding to each of the preset multiple information dissemination strategies; the experimental variable of the control experiment is whether or not to implement the information dissemination strategy to users in the target user group.
[0014] The strategy testing module is used to conduct the control experiment on the target user group and obtain the experimental result data corresponding to the target user group.
[0015] The first data processing module is used to determine the mean of the preset test statistic based on the experimental results data.
[0016] The second data processing module is used to calculate the variance of the preset test statistic based on the experimental result data using the resampling method.
[0017] The third data processing module is used to determine the confidence interval of the preset test statistic based on the mean and the variance.
[0018] The test result determination module is used to determine the test result of the interaction of the multiple information release strategies based on the confidence interval of the preset test statistic.
[0019] Thirdly, this application provides a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement a method for verifying the interaction of information publishing strategies as described in the first aspect.
[0020] Fourthly, this application provides a computer device including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement a method for verifying the interaction of information publishing strategies as described in the first aspect.
[0021] Fifthly, the present invention provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method for verifying the interaction of an information dissemination strategy as described in the first aspect.
[0022] The method, apparatus, equipment, and storage medium for the interactive operation of the inspection information release strategy provided in this application have the following technical effects:
[0023] The proposed solution provides a technical approach for testing the interaction between multiple information dissemination strategies. This solution calculates the mean and method of a preset test statistic based on the experimental results of multiple controlled experiments. Based on the confidence interval of the preset test statistic, it tests whether different information dissemination strategies have an interaction, measures the degree of mutual influence, and can more accurately evaluate and optimize information dissemination strategies. When calculating the variance of the preset test statistic, it is not necessary to consider whether the assumptions of the analysis of variance are met. Instead, it is calculated through a resampling method. The data that needs to be stored is the resampling data sample, which reduces the computational and storage difficulties.
[0024] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0025] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 This is a schematic diagram of the implementation environment for a method for verifying the interaction of information publishing strategies provided in an embodiment of this application;
[0027] Figure 2 This is a flowchart illustrating a method for verifying the interaction of information publishing strategies provided in an embodiment of this application;
[0028] Figure 3 This is a schematic diagram of a hierarchical traffic distribution mechanism provided in an embodiment of this application;
[0029] Figure 4 This is a schematic diagram of a construction process provided in an embodiment of this application;
[0030] Figure 5 This is a schematic diagram of a process for calculating the mean of a preset test statistic provided in an embodiment of this application;
[0031] Figure 6 This is a flowchart illustrating a method for calculating the variance of a preset test statistic, provided in an embodiment of this application.
[0032] Figure 7 This is a flowchart illustrating a method for determining test results provided in an embodiment of this application;
[0033] Figure 8This is a schematic diagram of a device for verifying the interaction of information publishing strategies provided in an embodiment of this application;
[0034] Figure 9 This is a schematic diagram of the hardware structure of a device for implementing a method for verifying the interaction of information publishing strategies, as provided in an embodiment of this application. Detailed Implementation
[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.
[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0037] To facilitate understanding of the technical solutions and their effects described in the embodiments of this application, the relevant technical terms are explained in the embodiments of this application:
[0038] A / B testing: An experimental method for testing the effectiveness of a strategy online. It involves randomly sampling the experimental sample into two mutually exclusive groups: an experimental group and a control group. The control group continues with the original strategy, while the experimental group uses the new strategy. The differences between the two groups are then compared to determine the effectiveness of the strategy.
[0039] Experimental group: The group that received policy treatment in the A / B test.
[0040] Control group: The group that did not receive strategy treatment in the A / B test.
[0041] Hypothesis testing, also known as statistical hypothesis testing, is a statistical inference method used to determine whether differences between samples or between a sample and the population are caused by sampling error or by inherent differences. Significance testing is the most commonly used method in hypothesis testing and a fundamental form of statistical inference. Its basic principle is to first make a certain hypothesis about the characteristics of the population, and then, through statistical reasoning from sampling studies, infer whether this hypothesis should be rejected or accepted.
[0042] Parametric testing, also known as parametric hypothesis testing, refers to statistical tests performed on parameters such as the mean and variance. Parametric testing is an important component of inferential statistics. When the population distribution is known (e.g., the population is normally distributed), statistical parameters of the population distribution are inferred based on sample data.
[0043] Nonparametric tests are an important component of statistical analysis methods, forming, along with parametric tests, the fundamental content of statistical inference. Nonparametric tests are methods for inferring the distribution of a population using sample data when the population variance is unknown or poorly known. Because nonparametric tests do not involve parameters related to the population distribution during the inference process, they are named "nonparametric" tests.
[0044] Please see Figure 1 This is a schematic diagram illustrating the implementation environment of a method for verifying the interaction of information publishing strategies provided in this application embodiment, such as... Figure 1 As shown, the implementation environment may include at least client 01 and server 02.
[0045] Specifically, the client 01 may include devices such as smartphones, desktop computers, tablets, laptops, digital assistants, smart wearable devices, monitoring devices, and voice interaction devices. It may also include software running on the device, such as web pages provided to users by service providers, or applications provided by those service providers. Specifically, the client 01 can be used to provide multiple preset information publishing strategies and receive verification results.
[0046] Specifically, the server 02 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server 02 may include network communication units, processors, and memory, etc. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. Specifically, the server 02 can be used to run an experimental platform to test the interaction between multiple information publishing strategies and obtain the test results.
[0047] This application embodiment can also be implemented using cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. It can also be understood as a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on cloud computing business models. Cloud technology requires cloud computing as its support. Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." Specifically, the server 02 and the database are located in the cloud. The server 02 can be a physical machine or a virtualized machine.
[0048] The following describes a method for examining the interaction of information dissemination strategies provided in this application. Figure 2 This is a flowchart illustrating a method for verifying the interaction of information publishing strategies, as provided in an embodiment of this application. This application provides the operational steps described in the embodiments or flowchart, but based on conventional or non-inventive methods, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Please refer to... Figure 2 The method for verifying the interaction of information publishing strategies provided in this application embodiment may include the following steps:
[0049] S210: Based on multiple preset information dissemination strategies, determine a control experiment corresponding to each of the information dissemination strategies; the experimental variable of the control experiment is whether or not the information dissemination strategy is implemented to users in the target user group.
[0050] In this embodiment, the information publishing strategy can also be called an information push strategy. Taking advertising push as an example, the push strategy can include the selection of advertising content, the design of the advertising interface, and the matching of advertising with users. For newly designed experimental strategies, online testing is required before official launch to measure the effectiveness of the new experimental strategy. Generally, an A / B experiment (also called A / B testing) is used to select the optimal strategy from the available options. An A / B experiment is a method for verifying hypotheses, and its core methods and principles are controlled experiments and hypothesis testing. In actual experiments, a sample of individuals is drawn from the population to form a sample unit, and the overall result is inferred from the individual experimental results. To ensure high availability of individual samples, multiple information publishing strategies will be in effect and function simultaneously on the experimental platform, that is, multiple A / B experiments will be conducted concurrently. Different experimental strategies may influence each other.
[0051] In this embodiment, to measure the degree of mutual influence, or interaction, between multiple experimental strategies, a control experiment is first designed for each information dissemination strategy. Since control experiments follow the principle of a single variable, causality can be discovered through comparison, and the degree of influence of the variable can be quantified based on the experimental results. In one control experiment of this embodiment, there are two groups: an experimental group and a control group. In the actual experiment, users are randomly assigned to either the experimental group or the control group. In this control experiment, the experimental variable is whether or not the information dissemination strategy is implemented for users; that is, the information dissemination strategy is implemented for users in the experimental group, while the information dissemination strategy is not implemented for users in the control group (i.e., the original strategy corresponding to the information dissemination strategy is implemented). In the control experiment, users can only receive processing from one strategy at a time. Therefore, when randomly assigning users, user groups with consistent or similar characteristics can be selected to minimize the impact of user differences on the experimental results. In determining the control experiment, it is also necessary to determine the experimental purpose, experimental indicators, and experimental type. For multiple information dissemination strategies, there are multiple corresponding control experiments. In the embodiments of this application, multiple control experiments can be implemented in a stratified manner, so that multiple control experiments can be implemented simultaneously and share the same user group.
[0052] S230: Conduct the control experiment on the target user group to obtain the experimental result data corresponding to the target user group.
[0053] In this embodiment of the application, the target user group is sampled from the total user population as a user sample for multiple control experiments conducted simultaneously. The experimental result data includes indicator data of multiple experimental metrics, such as exposure count, click count, click-through rate, conversion rate, etc. The experimental result data also includes strategy data, which indicates the strategy implemented for each user in the target user group. Since there are multiple control experiments, a strategy combination is implemented for each user. The strategy combination may include the number and specific content of the information release strategies that are hit.
[0054] In this embodiment of the application, the step of conducting the control experiment on the target user group to obtain the experimental result data corresponding to the target user group may include:
[0055] Based on the hierarchical and traffic-dividing mechanism, the control experiment is carried out on the target user group to obtain the experimental result data of each user in the target user group. The experimental result data includes the strategy data and the indicator data corresponding to the target indicators of each user.
[0056] It is understandable that a control experiment is a mutually exclusive experiment; a user cannot be simultaneously assigned to both the control and experimental groups. That is, only one of the information dissemination strategies can be implemented for that user: either the information dissemination strategy is applied or not. This user mutual exclusion ensures that the experimental results are not interfered with. When multiple control experiments are tested simultaneously, if these experiments are also mutually exclusive, the number of users who can participate in the experiment at the same time decreases, resulting in poor sample representativeness. If the next control experiment is conducted after the previous one has finished, the verification cycle increases, and efficiency decreases. Therefore, in this embodiment, a hierarchical distribution mechanism is used to segment the user group, making multiple control experiments orthogonal experiments (also known as hierarchical experiments). Each control experiment can use all users in the target user group. The hierarchical distribution mechanism mainly involves the concept of orthogonality. Taking two control experiments P and Q as an example, in an orthogonal experiment, if... Figure 3 As shown, there are two layers of experiments: the P experiment and the Q experiment. In the P experiment, users are divided into a control group Pa and an experimental group Pb, with each group comprising 50% of the users. In the Q experiment, users are also divided into a control group Qa and an experimental group Qb, with each group comprising 50% of the users. The so-called orthogonal experiment means that users in the control group Pa are evenly distributed into the control group Qa and the experimental group Qb in the Q experiment, and users in the experimental group Pb are also evenly distributed into the control group Qa and the experimental group Qb in the Q experiment. Figure 3 The squares shown only represent the orthogonality within a local sample range and do not represent all samples. In the embodiments of this application, a single control experiment is a mutually exclusive experiment, and multiple control experiments are orthogonal experiments.
[0057] It is understood that the control experiment in this application embodiment can discover the causal relationship between the implementation of the information release strategy and the target indicator by comparison, and can quantify the degree of influence of the information release strategy on the target indicator based on the experimental results data.
[0058] S250: Determine the mean of the preset test statistic based on the experimental results data.
[0059] In this embodiment of the application, hypothesis testing can be used to examine whether there is an interaction between multiple information dissemination strategies based on experimental results data. According to the general steps of hypothesis testing, firstly, a hypothesis about the characteristics of the population needs to be made; then, statistical inference is performed through sampling experiments; and finally, an inference is made as to whether the hypothesis should be rejected or accepted based on the inference results.
[0060] It is understood that, in the embodiments of this application, as Figure 4 As shown, the procedure further includes the following steps before step S250:
[0061] S510: Based on the aforementioned multiple information dissemination strategies, construct a hypothesis testing approach.
[0062] In the embodiments of this application, it is assumed that there is no interaction between the multiple information publishing strategies, that is, the common effect of the multiple information publishing strategies is the product of the effects of each information publishing strategy implemented individually.
[0063] Taking two information dissemination strategies as examples, consider the control experiments of any two standard traffic experiments. The strategies implemented by the groups in the two control experiments are denoted as A1, B1 and A2, B2, respectively. A1 is the information dissemination strategy implemented for the experimental group in the first control experiment, and B1 is the original strategy corresponding to A1 implemented for the control group in the first control experiment. A2 is the information dissemination strategy implemented for the experimental group in the second control experiment, and B1 is the original strategy corresponding to A1 implemented for the control group in the first control experiment. The standard traffic experiment refers to randomly sampling users from the target user group and assigning them to the control and experimental groups. Optionally, the number of users in the control and experimental groups is the same.
[0064] Table 1. Strategy combinations in the two control experiments
[0065]
[0066] As shown in the table above, 'a' represents the effect data produced by the combination of strategies A1 and A2, 'b' represents the effect data produced by the combination of strategies B1 and A2, 'c' represents the effect data produced by the combination of strategies A1 and B2, and 'd' represents the effect data produced by the combination of strategies B1 and B2. It is assumed that the two information dissemination strategies A1 and A2 in the two control experiments have no interaction, that is, it is assumed that the combined effect of A1 and A2 is the product of their individual effects, i.e.:
[0067] After simplification, we get:
[0068] Therefore, determining whether there is an interaction between two information dissemination strategies can be equivalent to a hypothesis testing problem, namely, testing H0: Whether this holds true, the alternative test is H1: If a decision to reject H0 is made, then it can be assumed that there is indeed an interaction between the two information dissemination strategies; otherwise, it can be assumed that there is no obvious interaction between the two information dissemination strategies.
[0069] Taking three information dissemination strategies as an example, there are two ways to test the interaction between them. One is to test each of the three strategies pairwise, with the proposed hypotheses described above, which will not be repeated here. The other is to test all three strategies together. For example, consider three controlled experiments. The strategies implemented by the groups in the three controlled experiments are denoted as A1 and B1, A2 and B2, and A3 and B3, respectively. Here, A1, A2, and A3 are the information dissemination strategies implemented for the experimental groups in the three controlled experiments, and B1, B2, and B3 are the corresponding original strategies. 'a' represents the effect data produced by implementing the three information dissemination strategies, b, c, and d represent the effect data produced by implementing two information dissemination strategies and one original strategy, respectively, e, f, and g represent the effect data produced by implementing one information dissemination strategy and two original strategies, respectively, and h represents the effect data produced by implementing all three original strategies. It is assumed that the common effect of the three information dissemination strategies is the product of the effects of each information dissemination strategy implemented individually, that is...
[0070] After simplification, we get:
[0071] Therefore, to determine whether there is an interaction between the three information dissemination strategies, one approach is to treat it as an equivalent hypothesis testing problem, i.e., testing H0: Whether this holds true, the alternative test is H1: If the decision to reject H0 is made, it can be assumed that there is indeed an interaction between the three information dissemination strategies; otherwise, it can be assumed that there is no obvious interaction between the three information dissemination strategies.
[0072] In this embodiment, the interaction between multiple information dissemination strategies can be tested pairwise, or multiple strategies can be tested together, although the latter is more complex. Preferably, testing the interaction between information dissemination strategies pairwise also facilitates targeted adjustments to the two interacting strategies after the test results are obtained.
[0073] S530: Construct a preset test statistic, which characterizes the degree of interaction between the multiple information dissemination strategies and the target indicator.
[0074] To make a decision on whether to accept or reject the hypothesis, a pre-defined test statistic needs to be constructed as a metric, according to the general steps of hypothesis testing. In this embodiment, a pre-defined test statistic is constructed based on the hypothesis. Taking the two control experiments mentioned above as examples, the expression for the pre-defined test statistic Y can be expressed as: The pre-defined test statistic can characterize the degree of interaction between the two information dissemination strategies on the target indicators indicated by a, b, c, and d. It can also be seen as the improvement in the effect of the two information dissemination strategies being implemented simultaneously relative to their individual implementation. The degree of improvement in effect is a proportional data and can be seen as following or approximately following a normal distribution.
[0075] S550: Set the confidence level or significance level parameter value for the test hypothesis.
[0076] According to the general steps of hypothesis testing, when formulating a hypothesis, a confidence level or significance level is usually set, such as setting a significance level α=0.05.
[0077] Understandably, in order to make a decision on whether to accept or reject the test hypothesis, according to the general steps of hypothesis testing, it is necessary to estimate the mean and variance of the preset test statistic. The data source is the experimental result data corresponding to the target user group obtained after the actual implementation of the experiment, so that the characteristic distribution of the population can be inferred through the sampling data.
[0078] Specifically, such as Figure 5 As shown, determining the mean of the preset test statistic based on the experimental results data may include the following steps:
[0079] S520: Determine multiple strategy combinations based on the multiple information publishing strategies.
[0080] In this embodiment of the application, multiple original strategies can be determined based on multiple information publishing strategies, and multiple information publishing strategies and corresponding original strategies can be orthogonally combined to obtain multiple strategy combinations, with each strategy combination having the same number of strategies; or multiple information publishing strategies can be directly combined to obtain strategy combinations, where the number of strategies in the strategy combinations is not the same, and only the information publishing strategies that are hit are included.
[0081] S540: Based on the strategy data of each user, the experimental result data is divided into a first experimental result data subset corresponding to each strategy combination.
[0082] It is understood that the experimental results data includes policy data, which represents the combination of policies implemented for each user. The policy data for each user in each subset of the first experimental results data is the same, while the policy data for users in different subsets of the first experimental results data are different. Therefore, the experimental results data is divided into multiple subsets based on different policy combinations.
[0083] S560: Calculate the average value of the target index corresponding to each strategy combination based on the index data corresponding to the target index in the first experimental result data subset.
[0084] S580: Determine the mean of the preset test statistic based on the average value of the target index corresponding to each of the strategy combinations.
[0085] In one specific embodiment of this application, the interaction between two information distribution strategies is examined. Based on the different combinations of strategies received by the target user group, the experimental results data can be divided into four subsets. The number of data samples in each of these four subsets, as well as the sum of indicators for each subset, such as total exposure, total clicks, and total cost, can be calculated. Thus, the average values of each indicator in these four subsets, such as average exposure, average clicks, and average cost, can be calculated. The mean of the preset test statistic is calculated using the average values of the target indicators in the four subsets. , , , We can obtain, denoted as It should be noted that the test of the interaction between multiple information dissemination strategies is conducted on a specific indicator, and the average value of the indicator obtained is data of the same indicator type.
[0086] S270: Based on the resampling method, the variance of the preset test statistic is calculated according to the experimental result data.
[0087] In this embodiment, the preset test statistic is a proportional data, and its variance formula is difficult to derive using parameter estimation. When using the delta method for variance estimation, the variance of the preset test statistic can be expressed as:
[0088]
[0089] in, and The variances of bc and ad are respectively, and further estimation using the difference method is needed, which is a complex and tedious process.
[0090] In this application, nonparametric estimation is used to calculate the variance of the pre-set test statistic. First, the target indicator data in the experimental results data is used as the data sample for data binning. Data binning is a data preprocessing technique. After binning, the average value of the target indicator is calculated based on the bin data. The calculation results are easy to store and expand, without needing to store the smallest granularity data sample, such as the experimental results data of a single user. Resampling refers to randomly sampling the original data sample, which can be with replacement, mainly including Bootstrap and Jackknife methods. Based on the original data... For example, five data points are sampled five times with replacement to obtain the Bootstrap sample. , And so on; while the number of samples in the Jackknife sample is 4, which can be: , , , , .
[0091] In the embodiments of this application, such as Figure 6 As shown, specifically, step S270 may include the following steps:
[0092] S610: The experimental result data is divided into data buckets to obtain multiple experimental result buckets.
[0093] S630: Resample the data from the plurality of experimental result buckets to obtain a plurality of second experimental result data subsets, wherein the second experimental result data subsets are data sets after removing any experimental result bucket data from the plurality of experimental result bucket data.
[0094] S650: Based on the plurality of second experimental result data subsets, determine the sample value of the preset test statistic corresponding to each second experimental result data subset.
[0095] S670: Variance estimation is performed based on the sample values of the preset test statistic to obtain the variance of the preset test statistic.
[0096] In one exemplary embodiment, the Jackknife resampling method is used. Specifically, the experimental results data of the user samples are randomly divided into n buckets, denoted as... The number of data points in each bucket can be different. When the sample size of the experimental data is very large, according to the law of large numbers, the number of data points in each bucket will tend to be equal. It should be noted that because the data is randomly allocated, the strategy combination accepted by the user corresponding to the data sample in each bucket can be different. The sample value of the pre-set test statistic Y is calculated after discarding data from one bucket each time, using the following formula:
[0097]
[0098] The calculation function f in the formula can be understood as follows: The remaining bucket data is merged into a subset, and the data samples in this subset are divided into multiple group datasets according to the different strategy combinations received by the user. The average value of the target indicator corresponding to each group dataset is calculated, and then a sample value of the preset test statistic Y is calculated from these average values. The process of calculating the average value of the indicator corresponding to each group dataset can be referred to as the process of calculating the mean of the preset test statistic described above. The difference is that the amount of data involved is different; in the process of calculating the sample value, data from any bucket is removed, while in the process of calculating the mean, all data are used. Furthermore, using the n sample values of the preset test statistic Y, the variance can be estimated as:
[0099]
[0100] in This is the mean of the preset test statistic obtained in the previous step.
[0101] S290: Determine the confidence interval of the preset test statistic based on the mean and the variance.
[0102] In this embodiment, based on the mean and variance of the preset test statistic, and combined with a pre-set confidence level or significance level, the confidence interval of the preset test statistic can be obtained. It is understood that the preset test statistic is a proportional data point, which can be considered to follow or approximately follow a normal distribution, thus determining the expression form of the confidence interval. For example, when the significance level is 0.05, the confidence interval of the preset test statistic can be expressed as:
[0103]
[0104] S2110: Determine the test result of the interaction of the multiple information release strategies based on the confidence interval of the preset test statistic.
[0105] In this application, confidence intervals are determined based on the mean and method of calculating a pre-set test statistic, and then the hypothesis is judged based on the confidence intervals. In other feasible embodiments, the hypothesis can also be judged by calculating the P-value. Essentially, the two methods are the same. The process of judging by confidence intervals in this application can be equivalently replaced by the process of judging by calculating the P-value, and this application does not limit this.
[0106] In the embodiments of this application, such as Figure 7 As shown, step S2110 may include the following steps:
[0107] S710: Determine the inspection threshold according to preset conditions; the preset conditions indicate that there is no interaction between the multiple information release strategies.
[0108] It is understood that the test threshold can be obtained based on the hypothesis or the expression of a preset test statistic. For example, if the hypothesis is that there is no interaction, then the test threshold is 0. Alternatively, a preset value can be used as the test threshold, such as 0.05.
[0109] S730: When the confidence interval includes the test threshold, it is determined that there is no interaction between the multiple information dissemination strategies.
[0110] S750: When the confidence interval does not contain the test threshold, it is determined that there is an interaction between the multiple information dissemination strategies.
[0111] In one specific embodiment of this application, as described above, a preset test statistic is used. Since the null hypothesis H0 states that there is no interaction, the test threshold is 0. By determining whether this confidence interval contains the value 0, we can determine whether there is an interaction between the two information dissemination strategies. If the confidence interval contains 0, such as (-1, 1), then there is no significant interaction between the two information dissemination strategies; if the confidence interval does not contain 0 and all values are greater than 0, such as (1, 3), then there is a significant positive interaction between the two information dissemination strategies; if the confidence interval does not contain 0 and all values are less than 0, such as (-3, -1), then there is a significant negative interaction between the two information dissemination strategies.
[0112] Please refer to Table 2, which shows some key data for testing the interaction between multiple information dissemination strategies (i.e., the experimental strategies in Table 2).
[0113] Table 2 Key data of the pre-set test statistic
[0114]
[0115] Experimental strategy A involves activating the media freshness feature, strategy B involves pre-filtering advertising materials, and strategy C involves abandoning support for the top-ranked e-commerce ad. It can be seen that the strategies in these three sets of experiments do indeed influence each other. Strategy A has significant positive and negative interactions with the other two strategies, respectively. Y1 and Y2 are pre-defined test statistics constructed for the specific metric of exposure rate. The positive interaction between the freshness feature and the material pre-filtering strategy indicates that ads that receive more exposure due to material pre-filtering are less likely to be controlled by freshness. Conversely, the negative interaction between the freshness feature and abandoning support for the e-commerce ad indicates that other ads that receive more exposure due to abandoning support for the e-commerce ad are more easily controlled by freshness, resulting in a significant negative exposure interaction between the freshness control strategy and these other strategies.
[0116] In this embodiment of the application, the obtained test results can also be displayed on the front end in the form of an experimental report, and the method can further include the following steps:
[0117] In response to a test result query command, a strategy interaction report is generated based on the test results;
[0118] Display the strategy interaction report.
[0119] In this application embodiment, the method can also be used as part of the productization of the experimental platform, displaying the test results in the form of an experimental report on the front end, or calculating the pairwise interactions between all control experiments offline (which can be understood as the interaction between control experiments being equivalent to the interaction between information dissemination strategies). The method provided in this application can indicate how many core indicators of the current experimental platform have significant pairwise interactions. Therefore, experiments with significant negative interactions can be identified in a timely manner, allowing researchers to modify their experimental strategies to reduce the negative impact of each other. Specifically, the parameters in the experimental strategies can be iteratively modified, or the parameters can be automatically modified according to predetermined rules. During the adjustment process, the magnitude of the interaction between the two experiments can be repeatedly calculated until the interaction between the two experiments is no longer significantly negative. It can also be combined with the volume release criteria of the experimental platform to avoid experiments that have a negative effect on the overall market after a large volume release is launched at the same time. The experimental report can also provide data on the overall impact of multiple experiments launched at the same time on the experimental platform. The overall impact on the platform is the combined effect of all experimental strategies online, but it is not a simple sum of the effects of individual experiments. The accurate measurement of interaction can be extended to the measurement of the overall effect of the platform, providing guidance for controlling the overall volume release rhythm.
[0120] Understandably, the solution provided in this application proposes a method for testing the interaction between multiple information dissemination strategies. This method calculates the mean and method of a preset test statistic based on the experimental results data of multiple controlled experiments. Based on the confidence interval of the preset test statistic, it tests whether there is an interaction between different information dissemination strategies, measures the degree of mutual influence, and can more accurately evaluate and optimize information dissemination strategies. When calculating the variance of the preset test statistic, it is not necessary to consider whether the assumptions of analysis of variance are met. Instead, it is calculated through a resampling method. The data that needs to be stored is the resampling data sample, which reduces the computational and storage difficulty and improves the testing efficiency.
[0121] This application embodiment also provides a device 800 for verifying the interaction of information publishing strategies, such as... Figure 8 As shown, the device 800 may include:
[0122] The experiment design module 810 is used to determine a control experiment corresponding to each of the preset multiple information dissemination strategies; the experimental variable of the control experiment is whether or not the information dissemination strategy is implemented to users in the target user group.
[0123] The strategy testing module 820 is used to conduct the control experiment on the target user group and obtain the experimental result data corresponding to the target user group.
[0124] The first data processing module 830 is used to determine the mean of the preset test statistic based on the experimental result data.
[0125] The second data processing module 840 is used to calculate the variance of the preset test statistic based on the experimental result data using the resampling method.
[0126] The third data processing module 850 is used to determine the confidence interval of the preset test statistic based on the mean and the variance.
[0127] The test result determination module 860 is used to determine the test result of the interaction of the multiple information release strategies based on the confidence interval of the preset test statistic.
[0128] In one embodiment of this application, the first data processing module 830 may include:
[0129] A strategy combination unit is used to determine multiple strategy combinations based on the multiple information publishing strategies;
[0130] The data partitioning unit is used to partition the experimental result data into a first subset of experimental result data corresponding to each strategy combination based on the strategy data of each user.
[0131] The first calculation unit is used to calculate the average value of the target indicator corresponding to each strategy combination based on the indicator data corresponding to the target indicator in the first experimental result data subset.
[0132] The second calculation unit is used to determine the mean of the preset test statistic based on the average value of the target index corresponding to each of the strategy combinations.
[0133] In one embodiment of this application, the second data processing module 840 may include:
[0134] The data bucketing unit is used to divide the experimental result data into multiple experimental result buckets.
[0135] The data resampling unit is used to resample the data of the plurality of experimental result buckets to obtain a plurality of second experimental result data subsets, wherein the second experimental result data subsets are data sets after removing any experimental result bucket data from the plurality of experimental result bucket data;
[0136] The third calculation unit is used to determine the sample value of the preset test statistic corresponding to each of the plurality of second experimental result data subsets based on the data.
[0137] The fourth calculation unit is used to estimate the variance based on the sample values of the preset test statistic, and obtain the variance of the preset test statistic.
[0138] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0139] This application provides a computer device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement a method for verifying the interaction of information publishing strategies as provided in the above method embodiments.
[0140] Figure 9 A schematic diagram of the hardware structure of a device for implementing a method for verifying the interaction of an information publishing strategy provided in an embodiment of this application is shown. This device may constitute or include the apparatus or system provided in the embodiment of this application. Figure 9As shown, device 9 may include one or more processors 902 (shown as 902a, 902b, ..., 902n in the figure) 902 (processor 902 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 904 for storing data, and a transmission device 906 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 9 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, device 9 may also include a... Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.
[0141] It should be noted that the aforementioned one or more processors 902 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the device 9 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0142] The memory 904 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method described in the embodiments of this application. The processor 902 executes various functional applications and data processing by running the software programs and modules stored in the memory 904, thereby realizing the above-mentioned method for verifying the interaction of information publishing strategies. The memory 904 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 904 may further include memory remotely located relative to the processor 902, and these remote memories can be connected to the device 9 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0143] The transmission device 906 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of device 9. In one example, the transmission device 906 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 906 may be a radio frequency (RF) module used for wireless communication with the Internet.
[0144] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of device 9 (or mobile device).
[0145] This application embodiment also provides a computer-readable storage medium, which can be disposed in a server to store at least one instruction or at least one program related to implementing a method for verifying the interaction of information publishing strategies in the method embodiment. The at least one instruction or at least one program is loaded and executed by the processor to implement the method for verifying the interaction of information publishing strategies provided in the above method embodiment.
[0146] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0147] This invention also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method for verifying the interaction of information dissemination strategies provided in the various optional embodiments described above.
[0148] As can be seen from the embodiments of the method, apparatus, medium and equipment for testing the interaction of information release strategies provided in this application,
[0149] The proposed solution provides a technical approach for testing the interaction between multiple information dissemination strategies. This solution calculates the mean and method of a preset test statistic based on experimental results from multiple controlled experiments. It then uses the confidence interval of the preset test statistic to test whether different information dissemination strategies interact, measuring the degree of mutual influence and enabling more accurate evaluation and optimization of information dissemination strategies. Furthermore, when calculating the variance of the preset test statistic, it eliminates the need to consider whether the assumptions of analysis of variance are met; instead, it uses a resampling method. The data that needs to be stored is the resampled data sample, reducing both computational and storage complexity.
[0150] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.
[0151] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0152] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for testing the interaction of information dissemination strategies, characterized in that, The method includes: Based on multiple preset information dissemination strategies, a control experiment is determined for each of the information dissemination strategies; the experimental variable of the control experiment is whether or not the information dissemination strategy is implemented to users in the target user group. The control experiment was conducted on the target user group to obtain the experimental result data corresponding to the target user group. Based on the experimental results data, the mean of the preset test statistic is determined; the preset test statistic characterizes the degree of interaction of the multiple information dissemination strategies with respect to the target indicator; Based on the resampling method, the variance of the preset test statistic is calculated from the experimental results data. Based on the mean and the variance, determine the confidence interval of the preset test statistic; Based on the confidence interval of the preset test statistic, determine the test results of the interaction of the multiple information release strategies; The step of calculating the variance of the preset test statistic based on the experimental results data using the resampling method includes: The experimental results data are binned to obtain multiple experimental results bins; The data from the multiple experimental result buckets are resampled to obtain multiple second experimental result data subsets. The second experimental result data subsets are the data sets after removing any data from the multiple experimental result buckets. Based on the plurality of second experimental result data subsets, determine the sample value of the preset test statistic corresponding to each second experimental result data subset; Variance estimation is performed based on the sample values of the preset test statistic to obtain the variance of the preset test statistic.
2. The method according to claim 1, characterized in that, The process of conducting the control experiment on the target user group to obtain experimental result data corresponding to the target user group includes: Based on the hierarchical distribution mechanism, the control experiment is carried out on the target user group to obtain the experimental result data of each user in the target user group. The experimental result data includes the strategy data and the indicator data corresponding to the target indicators of each user.
3. The method according to claim 2, characterized in that, The step of determining the mean of the preset test statistic based on the experimental results data includes: Multiple strategy combinations are determined based on the aforementioned multiple information dissemination strategies; Based on the strategy data of each user, the experimental result data is divided into a first subset of experimental result data corresponding to each strategy combination; The average value of the target indicator corresponding to each strategy combination is calculated based on the indicator data corresponding to the target indicator in the first experimental result data subset. The mean of the preset test statistic is determined based on the average value of the target indicator corresponding to each of the strategy combinations.
4. The method according to claim 1, characterized in that, Before determining the mean of the preset test statistic based on the experimental results data, the method further includes: Construct the preset test statistic.
5. The method according to claim 1, characterized in that, The step of determining the test results of the interaction of the multiple information dissemination strategies based on the confidence interval of the preset test statistic includes: A test threshold is determined based on preset conditions; the preset conditions indicate that there is no interaction between the multiple information release strategies. When the confidence interval includes the test threshold, it is determined that there is no interaction between the multiple information dissemination strategies; When the confidence interval does not include the test threshold, it is determined that there is an interaction between the multiple information dissemination strategies.
6. The method according to claim 1, characterized in that, The method further includes: In response to a test result query command, a strategy interaction report is generated based on the test results; Display the strategy interaction report.
7. A device for testing the interaction of information dissemination strategies, characterized in that, The device includes: The experiment design module is used to determine a control experiment corresponding to each of the preset multiple information dissemination strategies; the experimental variable of the control experiment is whether or not to implement the information dissemination strategy to users in the target user group. The strategy testing module is used to conduct the control experiment on the target user group and obtain the experimental result data corresponding to the target user group. The first data processing module is used to determine the mean of a preset test statistic based on the experimental results data; the preset test statistic characterizes the degree of interaction of the multiple information dissemination strategies with respect to the target indicator; The second data processing module is used to calculate the variance of the preset test statistic based on the experimental result data using the resampling method. The third data processing module is used to determine the confidence interval of the preset test statistic based on the mean and the variance. The test result determination module is used to determine the test result of the interaction of the multiple information release strategies based on the confidence interval of the preset test statistic. The second data processing module includes: The data bucketing unit is used to divide the experimental result data into multiple experimental result buckets. The data resampling unit is used to resample the data of the plurality of experimental result buckets to obtain a plurality of second experimental result data subsets, wherein the second experimental result data subsets are data sets after removing any experimental result bucket data from the plurality of experimental result bucket data; The third calculation unit is used to determine the sample value of the preset test statistic corresponding to each of the plurality of second experimental result data subsets based on the data. The fourth calculation unit is used to estimate the variance based on the sample values of the preset test statistic, and obtain the variance of the preset test statistic.
8. The apparatus according to claim 7, characterized in that, The first data processing module includes: A strategy combination unit is used to determine multiple strategy combinations based on the multiple information publishing strategies; The data partitioning unit is used to partition the experimental result data into a first experimental result data subset corresponding to each strategy combination based on the strategy data of each user in the target user group. The first calculation unit is used to calculate the average value of the target indicator corresponding to each strategy combination based on the indicator data corresponding to the target indicator in the first experimental result data subset. The second calculation unit is used to determine the mean of the preset test statistic based on the average value of the target index corresponding to each of the strategy combinations.
9. A computationally readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement a method for verifying the interaction of information dissemination strategies as described in any one of claims 1 to 6.
10. A computer device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement a method for verifying the interaction of information dissemination strategies as described in any one of claims 1 to 6.