Information delivery strategy testing method and device

Abstract

The present invention provides a method and device for testing an information delivery strategy, the method comprising: obtaining first delivery data of historical information delivery activities, the first delivery data comprising delivery information received by users in a random domain, using the first delivery data and the random domain to simulate the effects of a first delivery strategy and a second delivery strategy, when the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy, performing a real traffic experiment on the first delivery strategy, and when the effect corresponding to the first delivery strategy is not better than the effect corresponding to the second delivery strategy, not performing a real traffic experiment on the first delivery strategy. By simulating the effects of new strategies in advance, some new strategies whose improvement effects are not obvious enough are eliminated before the information delivery experiment, effectively reducing traffic waste.

Description

Technical Field

[0001] The present invention relates to deep learning technology, and in particular to a testing method and device for an information delivery strategy. Background Art

[0002] In the field of e-commerce, it is necessary to deliver some preferential information or activity information to users. A major difficulty is how to formulate an information delivery strategy so that the information can reach more target users and be used by them.

[0003] In order to obtain a better delivery strategy, it is necessary to try a variety of different delivery strategies and obtain a better strategy through AB testing methods. For example, in an information delivery process, users can be randomly divided into two groups, A and B, and then different delivery strategies are used to deliver information to users in group A and group B respectively. After the delivery is completed, the actual effects of group A and group B are compared to judge the pros and cons of the two delivery strategies. This type of technology is generally unable to effectively pre-judge the effectiveness of a new strategy before the experiment, but an information delivery experiment may require considerable engineering, operational and time costs. In order to solve this problem, some indirect numerical values ​​are generally used to capture the potential effects of new strategies.

[0004] However, this indirect method is always far from the online delivery experiment, so it is difficult to accurately judge the effectiveness of the new strategy before the experiment, resulting in a large amount of traffic, time and other resources being wasted in the information delivery experiment. Summary of the invention

[0005] The present invention provides a testing method and device for an information delivery strategy, which can eliminate some new strategies with insufficient improvement effects before an information delivery experiment, thereby effectively reducing traffic waste.

[0006] In a first aspect, the present invention provides a method for testing an information delivery strategy, comprising:

[0007] Acquire first delivery data of a historical information delivery activity, where the first delivery data is obtained by performing random delivery on users in a random domain, and the first delivery data includes delivery information received by users in the random domain;

[0008] Using the first delivery data and the random field, simulating the effects of the first delivery strategy and the second delivery strategy to obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy, wherein the first delivery strategy, the second delivery strategy and the historical information delivery activity use the same data;

[0009] When the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy, a real traffic experiment is performed on the first delivery strategy, and the experimental effect corresponding to the first delivery strategy is counted.

[0010] Optionally, a real traffic experiment is conducted on the first delivery strategy, including:

[0011] Calculating the unallocated budget of the current information delivery activity and the unallocated traffic of the current information delivery activity;

[0012] When the budget required by the first delivery strategy is less than the unallocated budget of the current information delivery activity, and the traffic required by the first delivery strategy is less than the unallocated traffic of the current information delivery activity, adding the domain corresponding to the first delivery strategy to the domain of the current information delivery activity, wherein each delivery strategy corresponds to one domain;

[0013] Allocate traffic to the domain of the current information delivery activity according to the unallocated traffic of the current information delivery activity;

[0014] Allocate a budget to the domain for the current campaign.

[0015] Optionally, using the first delivery data and the random field to simulate the effects of the first delivery strategy and the second delivery strategy to obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy include:

[0016] Initialize a user set X corresponding to a target delivery strategy, where the initial set of X is an empty set, and the target delivery strategy is the first delivery strategy or the second delivery strategy;

[0017] According to the category of the delivery information received by the users in the random domain, the users in the random domain are divided into |L| sets X1, X1, ...X |L| , |L| is the number of categories of the delivery information, X j represents the set of users who actually receive the information of the jth category, j = 1, ..., |L|;

[0018] Use the target delivery strategy to deliver information to the |L| sets;

[0019] According to the delivery results corresponding to the target delivery strategies in the |L| sets, each set in the |L| sets is divided into L subsets to obtain L*L subsets, and the subsets are represented by X j,k ,k=1...|L|,X j represents a set of users who actually receive information of the jth category and who are simulated to receive information of the kth category, wherein the simulated receipt of information of the kth category is obtained by information delivery using the target delivery strategy;

[0020] Update X to X = X ∪ X j，j , ∪ represents the union operation;

[0021] The effects corresponding to the user set X corresponding to the target delivery strategy are counted to obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy.

[0022] Optionally, allocating traffic to the domain of the current information delivery activity according to the unallocated traffic of the current information delivery activity includes:

[0023] Determine that the set of domains included in the current information delivery activity is D, where D includes at least two domains;

[0024] Calculate the current remaining flow U of the current information delivery activity on day t t+ ;

[0025] According to U t+ And the traffic prediction function y obtains the unallocated traffic u on day t t , where u t =y(t,U t+ );

[0026] From U t+ Randomly draw from |U t+ |-u t users, and obtain the traffic U to be allocated on day t t- ;

[0027] Assign U by weight t- get is the traffic allocated to domain d on day t, Requirements: v d represents the total flow required by domain d;

[0028] renew

[0029] Optionally, the allocating a budget to the domain of the current information delivery activity includes:

[0030] Determine the budget B that needs to be allocated on day t t ;

[0031] B t Assigned to each domain is the budget allocated to domain d on day t, Requirements:

[0032] renew B d is the allocated budget for domain d, b d is the total budget required for domain d.

[0033] Optionally, the calculation of the current remaining flow U of the current information delivery activity on day t t+ ,include:

[0034] Calculate U by the following formula t+ :

[0035] Among them, ∪ represents the union operation, U t Indicates the new traffic of the current information delivery activity on day t. The value range of t1 is 0 to t-1. Represents the new traffic of domain d on day t1.

[0036] Optional, y(t,U t+ ) satisfies the following properties:

[0037] y(t,U t+ )<y(t-1,U t+ );

[0038] y(t,U t+ )>0(t<T),y(T,U t+ )=0;

[0039] y(t,U t+ )<|U t+ |;

[0040] If |U t+ |<|U' t+ |, then y(t,U t+ )<y(t,U' t+ );

[0041] Where T represents the duration of the current information delivery activity, U' t+ Represents the current remaining traffic of the current information delivery activity on day t.

[0042] Optionally, the calculating of the unallocated budget of the current information delivery activity includes:

[0043] The unallocated budget B′ of the current information delivery activity is calculated using the following formula:

[0044]

[0045] B represents the total budget of the current information delivery activity, D represents the set of domains included in the current information delivery activity, and B d is the budget allocated to domain d, b d The budget required for domain d.

[0046] Optionally, the calculating the unallocated traffic of the current information delivery activity includes:

[0047] The unallocated traffic u′ of the current information delivery activity is calculated using the following formula:

[0048]

[0049] Among them, v C Indicates the total traffic of the current information delivery activity. is the traffic allocated to domain d on day t.

[0050] Optionally, the traffic of all domains included in the current information delivery activity has no intersection, and the traffic of each domain is homogeneous;

[0051] The sum of the budgets of all domains included in the current information delivery activity is equal to the total budget of the current information delivery activity, and the ratio of the budget and traffic of each domain is constant.

[0052] In a second aspect, the present invention provides a testing device for an information delivery strategy, comprising:

[0053] An acquisition module, configured to acquire first delivery data of a historical information delivery activity, wherein the first delivery data is obtained by performing random delivery on users in a random domain, and the first delivery data includes delivery information received by users in the random domain;

[0054] an effect simulation module, used to use the first delivery data and the random field to simulate the effects of the first delivery strategy and the second delivery strategy, and obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy, wherein the first delivery strategy, the second delivery strategy and the historical information delivery activity use the same data;

[0055] An effect comparison module, used to compare the effect corresponding to the first delivery strategy with the effect corresponding to the second delivery strategy;

[0056] The experimental module is used to perform a real traffic experiment on the first delivery strategy and to count the experimental effect corresponding to the first delivery strategy when the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy.

[0057] In a third aspect, the present invention provides an electronic device, comprising: at least one processor and a memory;

[0058] The memory stores computer-executable instructions;

[0059] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to the first aspect of the present invention.

[0060] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect of the present invention.

[0061] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method as described in the first aspect of the present invention is implemented.

[0062] The information delivery strategy testing method and device provided by the present invention obtains the first delivery data of the historical information delivery activities, the first delivery data includes the delivery information received by the user in the random domain, uses the first delivery data and the random domain to simulate the effects of the first delivery strategy and the second delivery strategy, and when the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy, the first delivery strategy is subjected to a real traffic experiment, and when the effect corresponding to the first delivery strategy is not better than the effect corresponding to the second delivery strategy, the first delivery strategy is not subjected to a real traffic experiment. By simulating the effect of the new strategy in advance, some new strategies whose improvement effect is not obvious enough are eliminated before the experiment, which effectively reduces the waste of traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0064] Figure 1 The overall framework diagram of the method provided by the present invention;

[0065] Figure 2 A flowchart of a method for testing an information delivery strategy provided in Embodiment 1 of the present invention;

[0066] Figure 3 is a schematic diagram of the simulation process;

[0067] Figure 4 A flow chart of the real flow experiment method provided in Example 2 of the present application;

[0068] Figure 5 A schematic diagram of traffic distribution;

[0069] Figure 6 This is an architecture diagram of the coupon distribution system;

[0070] Figure 7 A schematic diagram of the structure of a testing device for providing an information delivery strategy provided by an embodiment of the present invention;

[0071] Figure 8 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention.

[0072] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0073] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0074] The present invention provides a testing method for an information delivery strategy, which can be applied to an information delivery system. The information delivery system is used to deliver information to corresponding users according to the information delivery strategy, wherein the delivered information can be preferential information, subsidy information, electronic red envelopes, etc. of commodities.

[0075] Let C represent an information delivery activity, and define C = (V, B, T, L), where T is the number of days the activity lasts. The traffic of the information delivery activity is defined as the set of users covered by the activity V = (U0, U1, ..., U t , ...U T ), t=1..T, U0 is the initial user set of the information delivery activity, U t (t>0) is the set of new users on the tth day of the activity, satisfying That is, the newly added users in any two days will not overlap, or a user can only be added once in an information delivery activity.

[0076] Generally speaking, U0 is significantly greater than U t (t>0), use to represent the set of all users covered by the information delivery activity, and ∪ represents the union operation. B represents the budget of the information delivery activity, which is the total amount of information delivered. The form of the budget may vary depending on the delivery activity. For example, the budget can represent the discount amount, the number of coupons, the subsidy amount, the number of subsidy coupons, the number of red envelopes, the amount of red envelopes, etc.

[0077] L = {l k|k=1,...,K} represents a set of categories of delivery information. For example, when the delivery information is a coupon, L represents a set of coupon discount strengths, where K≥2, K is the total number of discount strengths. For example, if there are two types of coupons {10 off for purchases over 100, 20 off for purchases over 100}, then the value of K is 2.

[0078] For an information delivery campaign C, an information delivery strategy is defined as a function f that maps from a triple (u, b, L) to another function g, where Specifically, function f generates a coupon issuance function g = f(u, b, L) based on the candidate user set u and the partial budget b. g determines the category of the delivery information obtained by each user i∈u, l i =g(i),i∈u,l i ∈L.

[0079] In the prior art, an AB experiment is used to test the effects of two different information delivery strategies, and a better information delivery strategy is selected according to the effects of the information delivery strategies. However, each coupon delivery experiment requires a large amount of traffic and time.

[0080] In order to solve the problems of the prior art, the present invention provides a testing method for information delivery strategies. When a new strategy appears, the delivery effect of the new strategy can be simulated on historical random data. The method can use standards that are completely consistent with real experiments to pre-judge the possible effects of the new strategy. If the effect corresponding to the new strategy is significantly better than the effect corresponding to the existing optimal strategy, a real traffic experiment is conducted. If the effect corresponding to the new strategy is not better than the effect corresponding to the optimal strategy, the real traffic experiment is not conducted, thereby eliminating some delivery strategies that do not show obvious improvements before the experiment, effectively reducing traffic waste.

[0081] Figure 1 The overall framework diagram of the method provided by the present invention is as follows: Figure 1As shown, the information delivery system receives a new strategy, then reads the optimal strategy in the existing strategy library, simulates the effects of the new strategy and the optimal side, and when the effect corresponding to the new strategy is better than the effect corresponding to the optimal strategy, adds a new experiment, that is, adds the new strategy to the experiment of the current information delivery activity, which already includes at least one experiment. Then, the system performs traffic division and budget allocation. Traffic division is used to allocate the current remaining traffic of the current information delivery activity to the existing experiments (including the newly added experiment), and budget allocation is used to allocate the budget in the current information delivery activity to the existing experiments. After the budget allocation, information delivery is performed, which means that the budget allocated to each experiment is actually delivered to the traffic corresponding to the experiment. After all traffic and budget are allocated, the effect is statistically calculated, that is, the effect corresponding to each delivery strategy is statistically calculated, and finally, the strategy comparison is performed, that is, the effect corresponding to each delivery strategy is compared, and the optimal delivery strategy is selected.

[0082] This method supports the flexible insertion of new comparative experiments in longer information delivery activities, greatly increasing the utilization rate of online traffic. It can be understood that an experiment in an information delivery activity is a domain, and a domain can be understood as a sub-activity consisting of traffic and budget.

[0083] Figure 2 This is a flow chart of a method for testing an information delivery strategy provided in Embodiment 1 of the present invention, such as Figure 2 As shown, the method provided in this embodiment includes the following steps:

[0084] S101. Acquire first delivery data of a historical information delivery activity. The first delivery data is obtained by performing random delivery to users in a random domain. The first delivery data includes delivery information received by users in the random domain.

[0085] Assuming that there is no need to arrange a comparative experiment for a certain information delivery activity, then the activity C will usually be divided into two domains D = {E, R}: model domain E and random domain R. The traffic in the model domain E will be used for information delivery according to the current optimal delivery strategy, while the traffic in the random domain R will be randomly delivered to users according to the budget. In fact, regardless of whether a comparative experiment is arranged in a certain activity, a domain will always be used for random delivery, and the domain for random delivery is called the random domain.

[0086] The information delivery system collects random data and user conversion data of historical information delivery activities. The random data includes the label of each user in the random domain. Each user in the random domain has two labels, one label is used to indicate whether the delivery information has been received, and the other label is used to indicate whether an action associated with the delivery information has been performed. The action associated with the delivery information includes purchasing, placing an order, or adding to a shopping cart, etc.

[0087] The value of the label in the random data is empty, and the user conversion data is used to assign values ​​to the labels in the random data, or to label them. The user conversion data records which users have received the delivery information, and which users have purchased or ordered the products. Based on the random data and user conversion data, it can be obtained whether each user in the random domain has received the delivery information, and the category of the delivery information received.

[0088] S102: Use the first delivery data and the random domain to simulate the effects of the first delivery strategy and the second delivery strategy to obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy, wherein the first delivery strategy, the second delivery strategy and the historical information delivery activity use the same data.

[0089] Among them, the first delivery strategy can be a new strategy, and the second delivery strategy can be an existing optimal strategy. The two delivery strategies use the same data to simulate the effects. By comparing the corresponding effects of the two delivery strategies, it is determined which new strategy is better.

[0090] The first delivery strategy and the second delivery strategy may both be new strategies. By comparing the corresponding effects of the two delivery strategies, it is determined which new strategy is better.

[0091] Among them, the first delivery strategy, the second delivery strategy and the historical information delivery activities use the same data, which can be understood as the budget and traffic used by the first delivery strategy and the second delivery strategy are the same as the budget and traffic used by the historical information delivery activities.

[0092] Exemplarily, the simulation process includes the following steps:

[0093] (1) Initialize the user set X corresponding to the target delivery strategy. The initial set of X is an empty set.

[0094] The target delivery strategy is the first delivery strategy or the second delivery strategy, that is, the simulation process of the first delivery strategy and the second delivery strategy is the same.

[0095] (2) According to the categories of delivery information received by users in the random domain, the users in the random domain are divided into |L| sets X1,X1,...X |L| , |L| is the number of categories of information released, X j represents the set of users who actually receive the information of the jth category, j=1,...,|L|.

[0096] Among them, the category of delivery information received by users in the random domain is obtained from the first delivery data, and whether each user has received the delivery information and the category of the delivery information received can be obtained from the first delivery data. Assuming that there are three types of delivery information: first delivery information, second delivery information and third delivery information, then the value of |L| is 3, and the users in the random domain are divided into 3 sets: X1, X2 and X3, X1 represents the set of users who actually received the first delivery information, X2 represents the set of users who actually received the second delivery information, and X3 represents the set of users who actually received the third delivery information.

[0097] (3) Use the target delivery strategy to deliver information to |L| sets.

[0098] The first delivery strategy and the second delivery strategy both deliver information in the |L| sets, and use the same budget when delivering information, that is, the same budget is used to deliver information in the same traffic using different delivery strategies, and then the results are compared.

[0099] (4) According to the delivery results corresponding to the target delivery strategies in the |L| sets, each set in the |L| sets is divided into L subsets to obtain L*L subsets, which can be expressed as X j，k , k = 1...|L|, X j It represents the set of users who actually receive the information of the jth category and the set of users who are simulated to receive the information of the kth category, wherein the simulated receipt of the information of the kth category is obtained by using the target delivery strategy for information delivery.

[0100] The information actually received by the user refers to the information delivered to the user in the historical information delivery activities, and the information simulated to the user refers to the information delivered to the user when the first delivery strategy and the second delivery strategy are used to simulate the effect. When different delivery strategies are used to deliver information, the set of users reached by the information is different, and the set of users reached by delivery strategies and random delivery is also different.

[0101] (5) Update X to X = X∪X j，j , ∪ represents the union operation.

[0102] X j,j represents the set of users who have received information of the jth category both in reality and in simulation.

[0103] Figure 3 is a schematic diagram of the simulation process, such as Figure 3As shown in the figure, there are three types of delivery information: first delivery information, second delivery information and third delivery information. X1, X2 and X3 are random user sets obtained by random delivery. X1 represents the set of users who actually received the first delivery information, X2 represents the set of users who actually received the second delivery information, and X3 represents the set of users who actually received the third delivery information. Then, after using the target delivery strategy to deliver to users in X1, a total of three subsets are obtained: X 1,1 , X 1,2 and X 1,3 , after using the target delivery strategy to deliver to users in X2, we get three subsets: X 2,1 , X 2,2 and X 2,3 , after using the target delivery strategy to deliver to users in X3, we get three subsets: X 3,1 , X 3,2 and X 3,3 .

[0104] X 1,1 represents the set of users who actually and simulatedly received the first delivery information, X 1,2 represents the set of users who actually received the first delivery information but simulated the second delivery information. 1,3 Indicates the set of users who actually received the first delivery information but simulated the third delivery information. 2,2 represents the set of users who actually and simulatedly received the second delivery information, X 2,1 Indicates the set of users who actually received the second delivery information but simulated the first delivery information. 2,3 Indicates the set of users who actually received the second delivery information but simulated the third delivery information. 3,3 represents the set of users who actually and simulatedly received the third delivery information, X 3,1 represents the set of users who actually received the third delivery information but simulated the first delivery information. 3,2 It indicates the set of users who actually received the third delivery information but simulated the second delivery information.

[0105] From the subset {X 1,1 , X 1,2 , X 1,3 , X 2,1 , X 2,2 , X 2,3 , X 3,1 , X 3,2 , X 3,3}Select X 1,1 , X 2,2 and X 3,3The user set X corresponding to the target delivery strategy is formed. Through the above process, it can be found that the above simulation method has the following advantages: the user set X has the same delivery information distribution as the random user {X1, X2, X3}; the different user sets X generated by different delivery strategies are homogeneous. The homogeneity of the two sets means that the composition structure of the personnel in the two sets is the same, and the composition structure of the personnel can refer to the ratio of men and women, the ratio of age, the ratio of membership level, etc. Due to the above advantages of the simulation method, it is possible to have a fair preliminary judgment on the new strategy.

[0106] (6) Count the effects of the user set X corresponding to the target delivery strategy to obtain the effects of the first delivery strategy and the effects of the second delivery strategy.

[0107] The effect of the delivery strategy can be expressed by the conversion rate (CVR), which is: conversion rate = (number of users in X who have performed the action corresponding to the delivery information / total number of users in X) × 100%. The action corresponding to the delivery information can be the user's purchase behavior, and the total number of users in X refers to the total number of people to whom coupons are issued. Then the conversion rate = (number of users who purchased the product among the total number of people to whom coupons are issued / total number of people to whom coupons are issued) × 100%. The larger the conversion rate, the more users this information delivery activity can reach, and the better the delivery effect.

[0108] The effect of the delivery strategy can also be expressed or measured by the Gross Merchandise Volume (GMV), which is a transaction indicator for e-commerce platforms. It mainly refers to the total amount of orders placed, including paid and unpaid parts. GMV per capita = total amount of orders placed within the number of days of the information delivery activity / total number of people who received the delivery information during the information delivery activity. The larger the GMV per capita, the better the effect of this information delivery activity.

[0109] The effect of the delivery strategy can also be expressed by the incremental customer acquisition cost. The incremental customer acquisition cost is used to represent the delivery cost required to attract a certain number of customers. For example, When taking coupon distribution as an example, the allocated budget of the model group is the amount of coupons allocated to the model group, the traffic of the model group is the total number of users in the model group, and the model group is the domain for information delivery using the delivery strategy.

[0110] The random domain is also called the control group. In the embodiment of the present invention, when conducting an information delivery strategy experiment, the same random domain can be used. The coupon issuance method of the random domain is random, and the probability of each user in the random domain obtaining the delivery information is the same.

[0111] S103. When the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy, a real traffic experiment is conducted on the first delivery strategy to count the experimental effect corresponding to the first delivery strategy.

[0112] Compare the effect corresponding to the first delivery strategy with the effect corresponding to the second delivery strategy. When the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy, determine to conduct a real traffic experiment on the first delivery strategy. When the effect corresponding to the first delivery strategy is not better than the effect corresponding to the second delivery strategy, determine not to conduct a real traffic experiment on the first delivery strategy.

[0113] In this embodiment, the information delivery system can start a new information delivery activity to conduct a real traffic experiment on the first delivery strategy, or add a comparative experiment to the already started information delivery activity to conduct a real traffic experiment on the first delivery strategy. The method of adding a comparative experiment to the already started information delivery activity can refer to the description of the following embodiment 2, which will not be repeated here.

[0114] The experimental effect corresponding to the first delivery strategy can be expressed by, but is not limited to, conversion rate, per capita GMV or incremental customer acquisition cost.

[0115] After obtaining the effect corresponding to the first delivery strategy and the effect corresponding to the second delivery strategy, the information delivery system can display the effect corresponding to the first delivery strategy and the effect corresponding to the second delivery strategy to the user through a graphical user interface (GUI), and can also display the comparison results between the two to the user. For example, the comparison result is that the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy, which is convenient for users to understand the effects of different delivery strategies.

[0116] Optionally, when the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy, in addition to displaying the effect corresponding to the first delivery strategy and the effect corresponding to the second delivery strategy to the user, an experiment start option can also be displayed to the user, which is used for the user to choose whether to conduct a real traffic experiment.

[0117] After the user chooses to conduct a real traffic experiment, a real traffic experiment is conducted on the first delivery strategy. When the experimental effect corresponding to the first delivery strategy is obtained, the experimental effect is displayed to the user through the GUI, so that the user can understand the experimental effect corresponding to the delivery strategy.

[0118] Optionally, the parameters used in the above information delivery activities and the simulated effects will be stored for subsequent user inquiries. Similarly, the parameters of the information delivery activities of the real traffic experiment and the experimental effects will also be stored and queried.

[0119] The experimental effect corresponding to the first delivery strategy can be used by users to decide whether to conduct real information delivery. For example, when the experimental effect corresponding to the first delivery strategy is the incremental customer acquisition cost, if the incremental customer acquisition cost cannot meet the user's cost control needs, it can be decided not to deliver information. Alternatively, when the conversion rate or per capita GMV corresponding to the delivery strategy is low, it can be decided not to deliver information to avoid wasting resources.

[0120] In this embodiment, the first delivery data of the historical information delivery activity is obtained, and the first delivery data includes the delivery information received by the user in the random domain. The first delivery data and the random domain are used to simulate the effects of the first delivery strategy and the second delivery strategy. When the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy, the first delivery strategy is subjected to a real traffic experiment. When the effect corresponding to the first delivery strategy is not better than the effect corresponding to the second delivery strategy, the first delivery strategy is not subjected to a real traffic experiment. By simulating the effect of the new strategy in advance, some new strategies whose improvement effect is not obvious enough are eliminated before the experiment, which effectively reduces the waste of traffic.

[0121] In the embodiment of the present application, in order to flexibly insert comparative experiments during the information delivery activity, the domain division method is further designed. The traffic V in the random domain is divided into multiple sub-traffic, and then the budget B of the current information delivery activity is divided according to the weight of the sub-traffic to form a sub-activity, which is called a domain. Among them, the weight of the sub-traffic is pre-set, and the number of sub-traffic is the same as the number of delivery strategies, that is, one sub-traffic is allocated to one delivery strategy. Among them, the formal definition of the domain can be:

[0122] (1) Information delivery activity C can be divided into multiple domains D, where:

[0123] (2) The traffic of domain d is V d The following conditions are met:

[0124] a.

[0125] b.

[0126] c、v d and V d′ homogeneous.

[0127] (3) The budget of domain d is B d , B d The following conditions are met:

[0128] a、∑ d∈D B d = b;

[0129] b,

[0130] wherein, represents the traffic allocated to domain d on the t-th day or the newly added traffic on the t-th day, represents that the traffic of all domains under an activity is a non-overlapping partition of the traffic under the activity, and the traffic of each domain is homogeneous. Traffic homogeneity means that the composition structure of the people in each domain is the same. The composition structure of the people can refer to the male-female ratio, age ratio, membership level ratio, etc.

[0131] ∑ d∈D B d = b means that the total budget of all domains is equal to the total budget of the activity, represents that the ratio of the budget of each domain to the traffic scale of the domain is constant. The conditions in (2) and (3) above together ensure that different placement strategies are applied to different domains, and the final effects are comparable.

[0132] For a domain d, define b d as the budget required for domain d, and v d as the traffic required for domain d. The division of the budget is not difficult. The budget can be evenly distributed to each day, or the budget can be attenuated or increased by day. The main difficulty in traffic division lies in the fact that it is difficult to predict the situation of U t before the t-th day. To simplify the allocation process, in this application, the traffic is divided into two categories: allocated traffic and unallocated traffic. Among them, the allocated traffic refers to the traffic that is distributed to a certain domain through the domain management mechanism and is intervened by the placement strategy, while the traffic that has been obtained by the activity but has not been allocated to any domain is called unallocated traffic. By using the traffic prediction function to predict the possible total traffic v C = |U + | of activity C, and update this prediction over time.

[0133] To make full use of the traffic, a traffic prediction function y is introduced. y accepts a set of parameters (t, U t+ ), where t is the current time, t < T, T represents the duration of the current information placement activity, and U t+ is the current remaining traffic on the t-th day. Therefore, y returns the unallocated traffic u t , u t = y(t, U t+ ). y(t, U t+ ) satisfies the following four properties:

[0134] a, y(t, U t+ ) < y(t, U' t+ ).

[0135] b, y(t, Ut+ )>0(t<T),y(T,U t+ )=0.

[0136] c、y(t,U t+ )<|U t+ |.

[0137] d. If |U t+ |<|U' t+ |, then y(t,U t+ )<y(t-1,U t+ );.

[0138] Among them, property a is used to indicate that the unallocated flow gradually decreases with the increase of time. Property b indicates that the unallocated flow on day T is 0, that is, all the flow has been allocated, and the unallocated flow before day T is greater than 0. Property c indicates that the unallocated flow on day t is less than the current remaining flow on day t. Property d indicates that y decreases with |U t+ |Monotonically increasing, the more the current remaining traffic is, the more unallocated traffic will be.

[0139] Through the above four properties of ad, we can know that |U t+ |-u is the traffic that should be allocated on the tth day. y can be used to manage the undiverted traffic every day in a dynamic scenario. The nature of y determines that the unallocated traffic gradually decreases every day, and the remaining traffic on the last day of the activity is 0.

[0140] Based on the above domain description and traffic management, the following introduces the traffic allocation and budget allocation method for each domain in D on the tth day.

[0141] The traffic distribution of each domain in D on day t may include the following steps:

[0142] (1) Calculate the current remaining traffic U of the current information delivery activity on day t t+ .

[0143] Exemplarily, U is calculated as follows t+ : Among them, ∪ represents the union operation, U t represents the new traffic of the current information delivery activity on day t, ∪ t U t Indicates the total traffic of the current information delivery activity from the beginning to the tth day. The value range of t1 is 0 to t-1. represents the new traffic of domain d on day t1, represents the total traffic of the current information delivery activity from the beginning to the t-1th day. "\" represents the difference operation of the set. The result of A\B is the elements that exist in A but not in B. It means to remove the distributed traffic from all the traffic to get the remaining traffic.

[0144] (2) According to U t+ And the traffic prediction function y obtains the unallocated traffic u on day t t , where u t =y(t,U t+ ).

[0145] (3) From U t+ Randomly draw from |U t+ |-u t users, and obtain the traffic U to be allocated on day t t- , that is U t- =|U t+ |-u t .

[0146] (4) Allocate U by weight t- get is the traffic allocated to domain d on day t, Requirements: v d Represents the total flow required by domain d.

[0147] This step is to allocate the flow U on day t t- According to the weights assigned to each domain in D, It represents the ratio of the traffic allocated to domain d on day t to the traffic allocated to all domains on day t. It represents the ratio of the total traffic required by domain d to the total traffic required by all domains. If the ratios are the same, it means that the more total traffic a domain requires, the more traffic it will be allocated on day t. Conversely, the less total traffic a domain requires, the less traffic it will be allocated on day t.

[0148] (5) Update

[0149] After allocating traffic to domain d on the tth day, the total traffic required for domain d is updated, that is, the traffic allocated to domain d on the tth day is subtracted from the total traffic required for domain d. The total traffic required for domain d gradually decreases with the increase of activity time until the total traffic required for domain d is reduced to 0 on the last day of the activity.

[0150] The budget allocation for each domain in D on day t may include the following steps:

[0151] (1) Determine the budget B allocated on day t t .

[0152] The total budget of the current information delivery activity can be evenly distributed to each day, or a certain amount of budget can be increased or decreased based on the budget of the previous day to obtain the budget B allocated on the tth day.t .

[0153] (2) B t Assigned to each domain is the budget allocated to domain d on day t, Requirements:

[0154] This step is to allocate the budget B on day t. t Assigned to each domain in D, represents the ratio of the budget allocated to domain d on day t to the budget allocated to all domains on day t, It represents the ratio of the total traffic required by domain d to the total traffic required by all domains. The same ratio means that the budget allocated to a domain is directly proportional to the traffic allocated to the domain. That is, the more total traffic a domain requires, the more budget it will be allocated to on the tth day. Conversely, the less total traffic a domain requires, the less budget it will be allocated to on the tth day.

[0155] (3) Update B d is the allocated budget for domain d, b d is the total budget required for domain d.

[0156] Based on the above traffic, budget and domain management, this implementation can flexibly add a comparative experiment after the information delivery activity begins. Embodiment 2 of this application provides a real traffic experiment method. Figure 4 A flow chart of the real flow experiment method provided in Example 2 of the present application, such as Figure 4 As shown, the method of this embodiment includes the following steps:

[0157] S201. Calculate the unallocated budget and unallocated traffic of the current information delivery activity.

[0158] The current information delivery activity may include one or more domains. The following formula can be used to calculate the unallocated traffic u′ of the current information delivery activity at the tth time:

[0159]

[0160] Among them, v C Indicates the total traffic of the current information delivery activity. is the traffic allocated to domain d on day t, It represents the sum of the traffic allocated to all domains from day 1 to day t.

[0161] Exemplarily, the unallocated budget B′ of the current information delivery activity on day t is calculated using the following formula:

[0162]

[0163] B represents the total budget of the current information delivery activity, D represents the set of domains included in the current information delivery activity, and B d is the budget allocated to domain d, b d The budget required for domain d is, and it can be understood that the budget required for domain d changes dynamically. As time goes by, the budget required for domain d decreases gradually.

[0164] S202: When the budget required by the first delivery strategy is smaller than the unallocated budget of the current information delivery activity and the traffic required by the first delivery strategy is smaller than the unallocated traffic of the current information delivery activity, the domain corresponding to the first delivery strategy is added to the domain of the current information delivery activity.

[0165] The information delivery system determines whether the budget required by the first delivery strategy is less than the unallocated budget of the current information delivery activity, and determines whether the traffic required by the first delivery strategy is less than the unallocated traffic of the current information delivery activity. If the budget required by the first delivery strategy is less than the unallocated budget of the current information delivery activity, and the traffic required by the first delivery strategy is less than the unallocated traffic of the current information delivery activity, then the domain corresponding to the first delivery strategy is added to the domain of the current information delivery activity. If the budget required by the first delivery strategy is not less than the unallocated budget of the current information delivery activity, and / or the traffic required by the first delivery strategy is not less than the unallocated traffic of the current information delivery activity, then the domain corresponding to the first delivery strategy is refused to be added to the domain of the current information delivery activity.

[0166] S203: Allocate traffic to the domain of the current information delivery activity according to the unallocated traffic of the current information delivery activity, and allocate a budget to the domain of the current information delivery activity.

[0167] In this step, when allocating traffic and budget, the domain of the current information delivery activity includes the domain corresponding to the first delivery strategy, where the domain corresponding to the first delivery strategy is a newly added comparative experiment. Regardless of how many domains the current information delivery activity includes, the traffic allocation and budget allocation between each domain are performed independently, and the traffic of each domain does not overlap.

[0168] Figure 5 A schematic diagram of traffic distribution, such as Figure 5As shown, at the beginning of the information delivery activity, there are two domains in the information delivery activity: domain 1 and domain 2. On day t, the unallocated traffic on day t is predicted based on the current remaining traffic and the traffic prediction function. The current remaining traffic on day t minus the unallocated traffic on day t is obtained to obtain the traffic to be allocated on day t, and the traffic to be allocated on day t is allocated to domain 1 and domain 2 in proportion. The unallocated traffic on day t is used as the current remaining traffic on day t+1. On day t+1, domain 3 is newly added. The unallocated traffic on day t+1 is predicted based on the current remaining traffic on day t+1 and the traffic prediction function. The current remaining traffic on day t+1 minus the unallocated traffic on day t+1 is obtained to obtain the traffic to be allocated on day t+1. The traffic to be allocated on day t+1 is allocated to domain 1, domain 2, and domain 3 in proportion.

[0169] In this embodiment, by relying on a reliable traffic prediction model, it can be ensured that each domain obtains sufficient traffic and budget to support the comparative experiment in the domain before the end of the activity, and different domains are independent of each other and have no overlap in traffic. For domains that join the activity at the same time, the traffic of the two domains is homogeneous. Relying on the above-mentioned traffic management, comparative experiments can be flexibly added after the information delivery activity starts, thereby improving the utilization rate of online traffic.

[0170] Figure 6 This is an architecture diagram of the coupon distribution system, such as Figure 6 As shown in the figure, when users use shopping apps to shop, some user behavior data will be generated. Users may collect some coupons before shopping. User behavior data is stored in the user behavior data server. User behavior data includes user search behavior, browsing behavior, purchase behavior, coupon collection behavior, product return behavior, etc. The user behavior database obtains the user's coupon collection behavior and conversion behavior from the user behavior server. Conversion behavior refers to whether the user has made a purchase after receiving the coupon. The experimental database is used to store the results of AB experiments. The results of AB experiments include the coupon information collected by users in each experiment and the experiment to which the user belongs.

[0171] The strategy comparison module is used to obtain the user's coupon collection behavior and conversion behavior from the user behavior database, and obtain the experimental results from the experimental database. The effect of the coupon issuance strategy is obtained based on the experimental results, user behavior data and user data, and the effects of the coupon issuance strategies corresponding to the AB experiments are compared, and the optimal coupon issuance strategy is updated to the database.

[0172] The strategy update module is used to update the parameters of the coupon issuance strategy according to the data provided by the promotion sensitivity model to obtain a new strategy. The promotion sensitivity model is used to obtain some user parameters from the user behavior database, for example, the user's sensitivity to price.

[0173] When a new strategy is generated, the effect simulation module is used to simulate the effect of the new strategy using the method mentioned in the above embodiment. When the effect corresponding to the new strategy is better than the effect of the existing optimal strategy, it enters the implementation management module, and the implementation management module conducts experiments. The experiment management module obtains the duration of the coupon issuance activity, the budget of the coupon issuance activity, etc. from the coupon issuance activity database, and returns the traffic used for the coupon issuance activity from the traffic server, where the traffic server is used to obtain real traffic from the shopping app.

[0174] After obtaining the traffic, budget, duration and current status of the coupon issuance activity, the experiment management module performs domain management and traffic management according to the method mentioned in the above embodiment. The coupon allocation module obtains the traffic and budget allocated to each domain corresponding to the coupon issuance activity on the tth day from the domain management module and the traffic management module, and then allocates coupons to the traffic in the domain according to the strategy corresponding to each domain. The coupon issuance strategy used by the coupon allocation module is obtained from the strategy library, and the spark cluster is used to calculate the coupon allocation. The coupon issuance module is used to issue the allocated coupons to users in the shopping app.

[0175] Figure 7 A schematic diagram of a structure of a testing device for providing an information delivery strategy is provided in an embodiment of the present invention. Figure 7 As shown, the device 100 of this embodiment includes the following modules.

[0176] An acquisition module 11 is used to acquire first delivery data of a historical information delivery activity, wherein the first delivery data is obtained by performing random delivery on users in a random domain, and the first delivery data includes delivery information received by users in the random domain;

[0177] An effect simulation module 12 is used to use the first delivery data and the random field to simulate the effects of the first delivery strategy and the second delivery strategy to obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy, wherein the first delivery strategy, the second delivery strategy and the historical information delivery activity use the same data;

[0178] An effect comparison module 13, used to compare the effect corresponding to the first delivery strategy with the effect corresponding to the second delivery strategy;

[0179] The experimental module 14 is used to perform a real traffic experiment on the first delivery strategy and to count the experimental effect corresponding to the first delivery strategy when the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy.

[0180] Optionally, the test module 14 is specifically used for:

[0181] Calculating the unallocated budget of the current information delivery activity and the unallocated traffic of the current information delivery activity;

[0182] When the budget required by the first delivery strategy is less than the unallocated budget of the current information delivery activity, and the traffic required by the first delivery strategy is less than the unallocated traffic of the current information delivery activity, adding the domain corresponding to the first delivery strategy to the domain of the current information delivery activity, wherein each delivery strategy corresponds to one domain;

[0183] Allocate traffic to the domain of the current information delivery activity according to the unallocated traffic of the current information delivery activity;

[0184] Allocate a budget to the domain for the current campaign.

[0185] Optionally, the effect simulation module 12 is specifically used for:

[0186] Initialize a user set X corresponding to a target delivery strategy, where the initial set of X is an empty set, and the target delivery strategy is the first delivery strategy or the second delivery strategy;

[0187] According to the category of the delivery information received by the users in the random domain, the users in the random domain are divided into |L| sets X1, X1, ...X |L| , |L| is the number of categories of the delivery information, X j represents the set of users who actually receive the information of the jth category, j = 1, ..., |L|;

[0188] Use the target delivery strategy to deliver information to the |L| sets;

[0189] According to the delivery results corresponding to the target delivery strategies in the |L| sets, each set in the |L| sets is divided into L subsets to obtain L*L subsets, and the subsets are represented by X j,k ,k=1...|L|,X j represents a set of users who actually receive information of the jth category and who are simulated to receive information of the kth category, wherein the simulated receipt of information of the kth category is obtained by information delivery using the target delivery strategy;

[0190] Update X to X = X ∪ X j，j , ∪ represents the union operation;

[0191] The effects corresponding to the user set X corresponding to the target delivery strategy are counted to obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy.

[0192] Optionally, allocating traffic to the domain of the current information delivery activity according to the unallocated traffic of the current information delivery activity includes:

[0193] Determine that the set of domains included in the current information delivery activity is D, where D includes at least two domains;

[0194] Calculate the current remaining flow U of the current information delivery activity on day t t+ ;

[0195] According to U t+ And the traffic prediction function y obtains the unallocated traffic u on day t t , where u t =y(t,U t+ );

[0196] From U t+ Randomly draw from |U t+ |-u t users, and obtain the traffic U to be allocated on day t t -;

[0197] Assign U by weight t- get is the traffic allocated to domain d on day t, Requirements: v d represents the total flow required by domain d;

[0198] renew

[0199] Optionally, the allocating a budget to the domain of the current information delivery activity includes:

[0200] B t Assigned to each domain is the budget allocated to domain d on day t, Requirements:

[0201] renew B d is the allocated budget for domain d, b d is the total budget required for domain d.

[0202] Optionally, the calculation of the current remaining flow U of the current information delivery activity on day t t+ ,include:

[0203] Calculate U by the following formula t+ :

[0204] Among them, ∪ represents the union operation, U tIndicates the new traffic of the current information delivery activity on day t. The value range of t1 is 0 to t-1. Represents the new traffic of domain d on day t1.

[0205] Optional, y(t,U t+ ) satisfies the following properties:

[0206] y(t,U t+ )<y(t-1,U t+ );

[0207] y(t,U t+ )>0(t<T),y(T,U t+ )=0;

[0208] y(t,U t+ )<|U t+ |;

[0209] If |U t+ |<|U' t+ |, then y(t,U t+ )<y(t,U' t+ );

[0210] Where T represents the duration of the current information delivery activity, U' t+ Represents the current remaining traffic of the current information delivery activity on day t.

[0211] Optionally, the calculating of the unallocated budget of the current information delivery activity includes:

[0212] The unallocated budget B′ of the current information delivery activity is calculated using the following formula:

[0213]

[0214] B represents the total budget of the current information delivery activity, D represents the set of domains included in the current information delivery activity, and B d is the budget allocated to domain d, b d The budget required for domain d.

[0215] Optionally, the calculating the unallocated traffic of the current information delivery activity includes:

[0216] The unallocated traffic u′ of the current information delivery activity is calculated using the following formula:

[0217]

[0218] Among them, v C Indicates the total traffic of the current information delivery activity. is the traffic allocated to domain d on day t.

[0219] Optionally, the traffic of all domains included in the current information delivery activity has no intersection, and the traffic of each domain is homogeneous;

[0220] The sum of the budgets of all domains included in the current information delivery activity is equal to the total budget of the current information delivery activity, and the ratio of the budget and traffic of each domain is constant.

[0221] The device of this embodiment can be used to execute the method described in the above-mentioned embodiment 1 or embodiment 2. The specific implementation method and technical effects are similar and will not be described in detail here.

[0222] Figure 8 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention is shown in FIG. Figure 8 As shown, the electronic device 200 includes: a processor 21, a memory 22 and a transceiver 23, wherein the memory 22 is used to store instructions, the transceiver 23 is used to communicate with other devices, and the processor 21 is used to execute the instructions stored in the memory so that the electronic device 200 executes the method described in the above-mentioned embodiment 1 or embodiment 2. The specific implementation method and technical effect are similar and will not be repeated here.

[0223] Embodiment 5 of the present invention provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the method described in Embodiment 1 or Embodiment 2 above. The specific implementation method and technical effect are similar and will not be repeated here.

[0224] Embodiment 6 of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the method described in Embodiment 1 or Embodiment 2 above is implemented. The specific implementation method and technical effects are similar and will not be described in detail here.

[0225] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0226] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for testing an information delivery strategy, characterized in that: include: Acquire first delivery data of a historical information delivery activity, where the first delivery data is obtained by performing random delivery on users in a random domain, and the first delivery data includes delivery information received by users in the random domain; Using the first delivery data and the random domain, simulating the effects of the first delivery strategy and the second delivery strategy to obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy, wherein the first delivery strategy, the second delivery strategy and the historical information delivery activity use the same data; When the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy, a real traffic experiment is performed on the first delivery strategy to count the experimental effect corresponding to the first delivery strategy; Conduct a real traffic experiment on the first delivery strategy, including: Calculate the unallocated budget of the current information delivery campaign and the unallocated traffic of the current information delivery campaign; When the budget required by the first delivery strategy is less than the unallocated budget of the current information delivery activity, and the traffic required by the first delivery strategy is less than the unallocated traffic of the current information delivery activity, adding the domain corresponding to the first delivery strategy to the domain of the current information delivery activity, wherein each delivery strategy corresponds to one domain, and the traffic allocation and budget allocation between the domains are independent of each other; Allocate traffic to the domain of the current information delivery activity according to the unallocated traffic of the current information delivery activity; Allocate a budget to the domain for the current campaign.

2. The method according to claim 1, characterized in that The using the first delivery data and the random field to simulate the effects of the first delivery strategy and the second delivery strategy to obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy includes: Initialize a user set X corresponding to a target delivery strategy, where the initial set of X is an empty set, and the target delivery strategy is the first delivery strategy or the second delivery strategy; According to the category of the delivery information received by the users in the random domain, the users in the random domain are divided into |L| sets X1, X1, ...X |L| , |L| is the number of categories of the delivery information, X j represents the set of users who actually receive the information of the jth category, j = 1, ..., |L|; Use the target delivery strategy to deliver information to the |L| sets; According to the delivery results corresponding to the target delivery strategies in the |L| sets, each set in the |L| sets is divided into L subsets to obtain L*L subsets, and the subsets are represented by X j,k ,k=1...|L|,X j represents a set of users who actually receive information of the jth category and who are simulated to receive information of the kth category, wherein the simulated receipt of information of the kth category is obtained by information delivery using the target delivery strategy; Update X to X = X ∪ X j,j , ∪ represents the union operation; The effects corresponding to the user set X corresponding to the target delivery strategy are counted to obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy.

3. The method according to claim 1, characterized in that The allocating traffic to the domain of the current information delivery activity according to the unallocated traffic of the current information delivery activity includes: Determine that the set of domains included in the current information delivery activity is D, where D includes at least two domains; Calculate the current remaining flow U of the current information delivery activity on day t t+ ; According to U t+ And the traffic prediction function y obtains the unallocated traffic u on day t t , where u t =y(t,U t+ ); From U t+ Randomly draw from |U t+ |-u t users, and obtain the traffic U to be allocated on day t t- ; Assign U by weight t- get is the traffic allocated to domain d on day t, Requirements: v d represents the total flow required by domain d; renew 4. The method according to claim 3, characterized in that The allocating a budget to the domain of the current information delivery activity includes: Determine the budget B that needs to be allocated on day t t ; B t Assigned to each domain is the budget allocated to domain d on day t, Requirements: renew B d is the allocated budget for domain d, b d is the total budget required for domain d.

5. The method according to claim 3, characterized in that: The calculation of the current remaining flow U of the current information delivery activity on the tth day t+ ,include: Calculate U by the following formula t+ : Among them, ∪ represents the union operation, U t Indicates the new traffic of the current information delivery activity on day t. The value range of t1 is 0 to t-1. Represents the new traffic of domain d on day t1.

6. The method according to claim 3, characterized in that y(t,U t+ ) satisfies the following properties: y(t,U t+ )<y(t-1,U t+ ); y(t,U t+ )>0(t<T),y(T,U t+ )=0; y(t,U t+ )<|U t+ |; If |U t+ | < |U' t+ |, then y(t, U t+ ) < y(t, U' t+ ); Where T represents the duration of the current information delivery activity, U′ t+ Represents the current remaining traffic of the current information delivery activity on day t.

7. The method according to claim 1, characterized in that The calculation of the unallocated budget for the current information delivery activity includes: The unallocated budget B′ of the current information delivery activity is calculated using the following formula: B represents the total budget of the current information delivery activity, D represents the set of domains included in the current information delivery activity, and B d is the budget allocated to domain d, b d The budget required for domain d.

8. The method according to claim 1, characterized in that The calculating of the unallocated traffic of the current information delivery activity includes: The unallocated traffic u′ of the current information delivery activity is calculated using the following formula: Among them, v C Indicates the total traffic of the current information delivery activity. is the traffic allocated to domain d on day t.

9. The method according to any one of claims 1 to 5, characterized in that: The traffic of all domains included in the current information delivery activity has no intersection, and the traffic of each domain is homogeneous; The sum of the budgets of all domains included in the current information delivery activity is equal to the total budget of the current information delivery activity, and the ratio of the budget and traffic of each domain is constant.

10. A testing device for information delivery strategy, characterized in that: include: An acquisition module, configured to acquire first delivery data of a historical information delivery activity, wherein the first delivery data is obtained by performing random delivery on users in a random domain, and the first delivery data includes delivery information received by users in the random domain; an effect simulation module, used to use the first delivery data and the random field to simulate the effects of the first delivery strategy and the second delivery strategy, and obtain the effects corresponding to the first delivery strategy and the effects corresponding to the second delivery strategy, wherein the first delivery strategy, the second delivery strategy and the historical information delivery activity use the same data; An effect comparison module, used to compare the effect corresponding to the first delivery strategy with the effect corresponding to the second delivery strategy; An experiment module, configured to, when the effect corresponding to the first delivery strategy is better than the effect corresponding to the second delivery strategy, conduct a real traffic experiment on the first delivery strategy and count the experimental effect corresponding to the first delivery strategy; The experimental module is specifically used to calculate the unallocated budget of the current information delivery activity and the unallocated traffic of the current information delivery activity; When the budget required by the first delivery strategy is less than the unallocated budget of the current information delivery activity, and the traffic required by the first delivery strategy is less than the unallocated traffic of the current information delivery activity, adding the domain corresponding to the first delivery strategy to the domain of the current information delivery activity, wherein each delivery strategy corresponds to one domain, and the traffic allocation and budget allocation between the domains are independent of each other; Allocate traffic to the domain of the current information delivery activity according to the unallocated traffic of the current information delivery activity; Allocate a budget to the domain for the current campaign.

11. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images