Method, device, equipment and storage medium for determining information operation strategy

Abstract

The present invention provides a method, device, equipment and storage medium for determining an information operation strategy, the method comprising: obtaining the execution results of a plurality of object groups executing respective corresponding preset information operation strategies; obtaining at least one operation data corresponding to each object group based on the execution results; classifying at least one operation data corresponding to each object group based on the mutual relationship between the operation data, obtaining at least one operation data set corresponding to each object group; summarizing each operation data in each operation data set corresponding to each object group, obtaining summary information of each operation data set corresponding to each object group; resampling the summary information corresponding to each object group as new data, calculating the statistical significance results between each object group according to the hypothesis test principle, thereby determining the target information operation strategy. The present invention reduces the statistical deviation caused by the non-independence between data and improves the accuracy of determining the information operation strategy.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and specifically relates to an information operation strategy determination method, device, equipment and storage medium. Background Art

[0002] The AB experiment system is an experimental statistical analysis method based on the normal distribution. It assumes that the user's behavior data is independent and identically distributed, and then compares the means of the experimental group and the control group based on the central limit theorem to draw conclusions. When calculating the statistical significance of the difference between groups A and B, the prior art first collects the user behavior data of groups A and B. Then it is assumed that the user behaviors of groups A and B do not affect each other, and the sample data are independent, that is, it is assumed that the sample means and variances of groups A and B exist, and the samples obey independent and identical distributions. Then, the hypothesis testing principle can be used to calculate the statistical data of groups A and B, such as p-value, statistical power, etc., according to the proof by contradiction.

[0003] The existing technology is based on the premise that each user behavior is independent of each other. On this basis, the covariance between each user behavior is zero. However, in actual applications, each user behavior is not independent. If the covariance between non-independent data is ignored, the calculated data will be missing some values, that is, the variance will be underestimated, resulting in inaccurate experimental conclusions. This will cause the business side to conclude that the B group solution is better than the A group solution, but the actual B group solution will not achieve the expected effect after it is launched. Summary of the invention

[0004] In order to reduce the statistical deviation caused by non-independence between data, improve the accuracy of statistical significance conclusions of AB experiments, and thus determine the best information operation strategy suitable for the object, the present invention proposes an information operation strategy determination method, device, equipment and storage medium.

[0005] In one aspect, the present invention provides a method for determining an information operation strategy, the method comprising:

[0006] Obtaining the execution results of the preset information operation strategies corresponding to the multiple object groups;

[0007] Based on the execution result, obtaining at least one operation data corresponding to each object group;

[0008] Classifying at least one operation data corresponding to each object group based on the mutual relationship between the operation data to obtain at least one operation data set corresponding to each object group;

[0009] Summarize each operation data in each operation data set corresponding to each object group to obtain summary information of each operation data set corresponding to each object group;

[0010] Performing a resampling process with replacement on the summary information of each operation data set corresponding to each object group to obtain a resampling result corresponding to each object group;

[0011] Based on the resampling results corresponding to each object group, a normal distribution model corresponding to each object group is established;

[0012] According to the hypothesis testing principle and the normal distribution model corresponding to each object group, the statistically significant results between each object group are obtained;

[0013] According to the statistical significance results between each object group, a target information operation strategy is determined from preset information operation strategies corresponding to each of the multiple object groups.

[0014] On the other hand, the present invention provides an information operation strategy determination device, the device comprising:

[0015] An execution result acquisition module is used to obtain the execution results of the preset information operation strategies corresponding to the execution of the multiple object groups;

[0016] an operation data determination module, configured to obtain at least one operation data corresponding to each object group based on the execution result;

[0017] an operation data set determination module, configured to classify at least one operation data corresponding to each object group based on the mutual relationship between the operation data, and obtain at least one operation data set corresponding to each object group;

[0018] A summary information determination module, used to summarize each operation data in each operation data set corresponding to each object group to obtain summary information of each operation data set corresponding to each object group;

[0019] A resampling module is used to perform a resampling process with replacement on the summary information of each operation data set corresponding to each object group to obtain a resampling result corresponding to each object group;

[0020] Establishing a module, for establishing a normal distribution model corresponding to each object group based on the resampling result corresponding to each object group;

[0021] A statistically significant result determination module is used to obtain the statistically significant results between each object group based on the hypothesis testing principle and the normal distribution model corresponding to each object group;

[0022] The target information operation strategy determination module is used to determine the target information operation strategy from the preset information operation strategies corresponding to each of the multiple object groups according to the statistical significance results between each object group.

[0023] On the other hand, the present invention proposes a device, comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the information operation strategy determination method as described above.

[0024] On the other hand, the present invention proposes a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the information operation strategy determination method as described above.

[0025] The present invention proposes a method, device, equipment and storage medium for determining an information operation strategy. For each object group, the operation data with mutual relations therein are classified into the same operation data set, and then the operation data in each operation data set are summarized to obtain the summary information of each operation data set, so that each operation data set is independent of each other. Then, each operation data set is used as a new "user" to perform resampling processing on the new independent "users". Then, according to the hypothesis testing principle, the statistical data between each object group is calculated to obtain the statistical significance results between each object group. Finally, according to the statistical significance results between each object group, the target information operation strategy is determined from the preset information operation strategies corresponding to each of the multiple object groups, thereby reducing the variance underestimation problem caused by non-independent samples, greatly reducing the statistical deviation caused by the non-independence between data, and improving the accuracy of the statistical significance conclusion of each object group, so as to accurately determine the target information operation strategy that best suits the object, so that the object can provide the effect of information operation when executing the target information operation strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is an overall architecture diagram for determining the information operation strategy provided by an embodiment of the present invention.

[0028] Figure 2 yes Figure 1 The structural block diagram of the indicator analysis system in .

[0029] Figure 3It is a flow chart of the information operation strategy determination method provided by an embodiment of the present invention.

[0030] Figure 4 This is another flowchart of the information operation strategy determination method provided by an embodiment of the present invention.

[0031] Figure 5 This is another flowchart of the information operation strategy determination method provided by an embodiment of the present invention.

[0032] Figure 6 It is a logic block diagram of the information operation strategy determination method provided by an embodiment of the present invention in an application scenario.

[0033] Figure 7 It is a structural diagram of an information operation strategy determination device provided by an embodiment of the present invention.

[0034] Figure 8 It is a schematic diagram of a server structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0037] Figure 1It is an overall architecture diagram of the information operation strategy determination provided by the embodiment of the present invention, and the overall architecture can be used as the implementation environment of the information operation strategy determination. The overall architecture can include a World Wide Web (Web) system, a traffic management system, an experimental access system, an indicator analysis system, etc. Among them, the Web system can further include experimental reports, authority management, indicator configuration, data visualization, experimental configuration, etc., the traffic management system can further include various types of traffic management tools, and the experimental access system can further include experimental access and traffic screening, etc.

[0038] Continue as Figure 1 like Figure 2 As shown, the indicator analysis system can further include an indicator management module, a task scheduling module and an indicator analysis module. The indicator management module can include indicator definition, indicator and experiment management, and indicator and model association, etc. The task scheduling module is used to output indicator data, such as article click-through rate, etc. The indicator analysis module is used to perform statistical analysis on the indicator data to obtain experimental conclusions, such as whether to launch a certain plan or strategy, etc.

[0039] It should be noted that Figure 1 and Figure 2 Just an example.

[0040] The following introduces the basic concepts involved in the embodiments of this specification:

[0041] AB testing: Develop two (A / B) or more (A / B / n) plans for the same goal. In the same time dimension, allow visitor groups (target populations) with the same (similar) composition to randomly access these plans, collect user experience data and business data from each group, and finally analyze and evaluate the best plan, and officially adopt the evaluated plan.

[0042] Hypothesis testing: Hypothesis testing is a method used in inferential statistics to test statistical hypotheses. The basic idea of ​​hypothesis testing is the idea of ​​small probability proof by contradiction. The idea of ​​small probability means that small probability events (P<0.01 or P<0.05) basically will not occur in an experiment. The idea of ​​proof by contradiction is to first propose a hypothesis, such as assuming that there is no difference in the experimental effects of groups A and B, and then use appropriate statistical methods to determine the possibility of the hypothesis being true. If the possibility is small, the hypothesis is considered to be false, and the proof is that there is a significant difference in the experimental effects of groups A and B. If the possibility is large, the hypothesis is considered to be true.

[0043] Statistical significance: The level of risk required to reject the null hypothesis when it is true, also called the probability level or significance level.

[0044] p-value: A parameter used to determine the results of a hypothesis test, that is, the probability of the sample observation results or more extreme results when the null hypothesis is true. If the P-value is very small, it means that the probability of the null hypothesis is very small. If it occurs, according to the principle of small probability, there is reason to reject the null hypothesis. The smaller the P-value, the more sufficient the reason to reject the null hypothesis. In short, the smaller the P-value, the more significant the result. However, whether the test result is "significant", "moderately significant" or "highly significant" needs to be determined based on the size and value of the P-value.

[0045] Relative difference confidence interval: an interval with the upper and lower confidence limits of the statistic as the upper and lower bounds respectively.

[0046] Figure 3 It is a flow chart of an information operation strategy determination method provided by an embodiment of the present invention, and the information operation strategy determination method can be run in the above-mentioned overall architecture of information operation strategy determination. This specification provides method operation steps as described in the embodiment or flow chart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 3 As shown, the method may include:

[0047] S101. Obtain execution results of the preset information operation strategies corresponding to the respective ones of the multiple object groups.

[0048] In the embodiments of the present specification, objects, namely users, may be divided into multiple object groups first, and then each object group may be made to execute a different preset information operation strategy, thereby obtaining different execution results.

[0049] In a feasible embodiment, the preset information operation strategy may be an information click strategy, such as a video click strategy, an article click strategy, and the like.

[0050] S103. Based on the execution result, obtain at least one operation data corresponding to each object group.

[0051] In the embodiment of the present specification, the behavior data of each user in the object group, that is, the operation data of each user, such as video click data, article click data, etc., can be obtained according to the above execution results.

[0052] For example, in S101, users are divided into two object groups, A and B. The preset information operation policy corresponding to object group A is "the background of the button clicked for the video is red", and the preset information operation policy corresponding to object group B is "the background of the button clicked for the video is green". Then, users in object group A can watch the video by clicking the red button, and the operation data of users in object group A watching the video by clicking the button with a red background can be collected. Similarly, users in object group B can watch the video by clicking the green button, and the operation data of users in object group B watching the video by clicking the button with a green background can be collected.

[0053] In the embodiment of the present specification, since each object group includes at least one object, ie, a user, and each user may have multiple operation data, each object group will eventually correspond to at least one operation data.

[0054] S105. Classify at least one operation data corresponding to each object group based on the mutual relationship between the operation data to obtain at least one operation data set corresponding to each object group.

[0055] In practical applications, the formula for calculating the variance D(X) of a random variable X can be shown as formula (1):

[0056]

[0057] Among them, x1, x2, x3, ..., xn represent each sample. The above formula (1) calculates the variance of the experimentally observed samples x1, x2, x3, ..., xn, which will eventually be equal to the sum of the covariance Cov(xi, xj) between each random variable xi and other random variables xj. When it is assumed that the behavior of each user is independent, that is, the observed user behavior data xi, xj are independent of each other, the samples conform to the property of independent and identical distribution. Under this assumption, when i is not equal to j, the covariance Cov(xi, xj) between xi, xj will be equal to 0, and then formula (2) will be obtained:

[0058]

[0059] However, in actual applications, not all behaviors between users are independent. When calculating the variance in the above formula (1), the covariance Cov(xi,xj) of different samples cannot be ignored. If ignored, the calculated data will be missing some values, that is, the variance will be underestimated, resulting in inaccurate experimental conclusions. The consequence is that the business side concludes that strategy B is better than strategy A in the experiment, but strategy B has no effect when it is actually launched.

[0060] In order to avoid the problem of underestimated variance due to non-independence between data, reduce the statistical deviation caused by non-independence between data, and improve the accuracy of statistical significance conclusions for each object group, so as to determine the best information operation strategy that best suits the user, in the embodiment of this specification, operation data with mutual relations can be classified into the same set, so that each operation data set obtained by classification is independent of each other.

[0061] In a feasible embodiment, all operation data of the same user can be classified into the same operation data set according to the user's identification information, such as the user's identity number (ID). Since each operation data set corresponds to a different user, each operation data set is independent of each other, thereby solving the problem of non-independence between user behaviors. Specifically, Figure 4 As shown, the classifying the at least one operation data corresponding to each object group based on the mutual relationship between the operation data to obtain at least one operation data set corresponding to each object group may include:

[0062] S1051. Obtain identification information of the object to which each operation data corresponding to each object group belongs.

[0063] S1053. Aggregate the operation data of the objects in each object group that have the same identification information to obtain at least one operation data set corresponding to each object group.

[0064] Theoretically, the operation data of a user is divided into the same operation data set. If there are n users, there will be n operation data sets accordingly. However, the storage and calculation required for resampling according to a single user dimension are very huge. Based on this, in order to reduce the amount of system calculation and relieve system pressure, in a feasible embodiment, when the number of operation data sets is greater than the first threshold, any number of operation data sets can be merged, so that each operation data set will correspond to at least one user, thereby controlling the number of operation data sets within a preset range, such as 50, 100 or 150 operation data sets. It should be noted that the more operation data sets there are, the higher the calculation accuracy, but the corresponding amount of calculation will also be greater.

[0065] Specifically, after aggregating the operation data of the objects with the same identification information in each object group to obtain at least one operation data set corresponding to each object group, the method may further include:

[0066] Determine whether there is an object group whose number of operation data sets is greater than a first threshold;

[0067] If there is an object group whose number of operation data sets is greater than the first threshold, merging any number of operation data sets in the object group to obtain a merged operation data set corresponding to the object group;

[0068] The merged operation data set corresponds to at least one target object.

[0069] In a feasible embodiment, the operation data with mutual relationship can also be of the same type, that is, the operation data are divided into different sets according to the type of the operation data. For example, the operation data corresponding to object group A is: <User 1, funny videos, with clicks>, <User 1, current affairs related videos, no clicks>, <User 2, funny related videos, with clicks>, <User 3, funny related videos, with clicks>, then <User 1, funny videos, with clicks>, <User 2, funny related videos, with clicks>, and <User 3, funny related videos, with clicks> can be divided into the same set.

[0070] S107. Summarize each operation data in each operation data set corresponding to each object group to obtain summary information of each operation data set corresponding to each object group.

[0071] In the embodiments of this specification, since each operation data set is independent of each other, each operation data in each operation data set can be summarized separately, and the summarized data is used as new user operation data, and statistical calculations are performed on the new user information. Specifically, Figure 4 As shown, the operation data in each operation data set corresponding to each object group are summarized to obtain summary information of each operation data set corresponding to each object group, which may include:

[0072] S1071. Merge the objects to which the respective operation data in the respective operation data sets corresponding to each object group belong, to obtain the merged objects corresponding to the respective operation data sets in each object group.

[0073] S1073. Accumulate the operation data of the merged objects corresponding to each operation data set in each object group to obtain summary information corresponding to each operation data set in each object group.

[0074] In S1071, all users corresponding to each operation data set are merged into a new user, and the operation data of the merged users are accumulated in S1073. For example, the operation data in one of the operation data sets in object group A is as follows: <user 1, video 1, clicked>, <user 1, video 2, no click>, <user 5, video 2, clicked>, <user 5, video 3, clicked>, then user 1 and user 5 are merged into a new user 1 through S1071, and all video numbers and click times in the operation data set are accumulated, and the summary information corresponding to the operation data set is obtained as <new user 1, 4, 3>.

[0075] S109. Perform resampling with replacement on the summary information of each operation data set corresponding to each object group to obtain a resampling result corresponding to each object group.

[0076] In the embodiments of the present specification, after the summary information of each operation data set is obtained, a resampling process may be performed on the summary information, and the resampling process includes but is not limited to a jackknife method or a bootstrap method.

[0077] The following uses jackknife as an example to introduce the resampling process of the summary information of each operation data set:

[0078] Jackknife is a resampling method. Through experiments, sample data x of user behavior can be collected. Although the true distribution F of the user behavior cannot be known, the sample information can be used to estimate the parameter s_hat of the population. At this time, the deviation between the sample estimated parameter s_hat and the true value of the population parameter is required, but since the population distribution is unknown, this deviation is also unknown. The Jackknife method uses sample data x as the empirical distribution F0, performs sampling with replacement to obtain the resampling result x*, and then calculates s*_hat based on the resampling result x*. In the video click-through rate scenario, this s*_hat can be the video click-through rate. In theory, after multiple samplings according to the central limit theorem, the deviation between the average value of s*_hat calculated by resampling and the true parameter of the empirical distribution F0 should be close to the deviation between s_hat and the true value of the population parameter. Therefore, the deviation between s_hat and the true value of the population parameter can be replaced by the deviation between the average value of s*_hat and the true parameter of the empirical distribution F0.

[0079] The basic principle of Jackknife is that each time it resamples, it gets a new set of samples by removing the same thing in x. For example, if the original sample is x = (x1, x2, x3, ..., xn), then in the i-th round of resampling, the resampled sample obtained is x(i) = (x1, x2, x3, ..., x{i-1}, x{i+1}, ... xn).

[0080] Specifically, performing resampling with replacement on the summary information of each operation data set corresponding to each object group to obtain a resampling result corresponding to each object group may include:

[0081] Perform N resampling processes on the summary information of each operation data set corresponding to each object group to obtain N resampling sets corresponding to each object group;

[0082] Wherein, N is the number of the summary information, each resampling set includes N-1 summary information, and the i-th resampling set is a set that does not include the i-th summary information, 1≤i≤N, i and N are both positive integers.

[0083] S1011. Based on the resampling results corresponding to each object group, a normal distribution model corresponding to each object group is established.

[0084] In the embodiment of this specification, since each operation data set has no correlation, that is, each operation data set is independent and identically distributed, the central limit theorem can be correctly applied to establish a normal distribution model for these operation data sets. Specifically, Figure 5 As shown, the establishing of a normal distribution model corresponding to each object group based on the resampling result corresponding to each object group may include:

[0085] S10111. Determine the mean and variance of the information operation corresponding to each object group based on the summary information in the N resampling sets corresponding to each object group.

[0086] S10113. Establish a normal distribution model corresponding to each object group based on the mean and variance of the information operation corresponding to each object group.

[0087] S1013. According to the hypothesis testing principle and the normal distribution model corresponding to each object group, obtain the statistical significance results between each object group.

[0088] In the examples of this specification, statistically significant results include but are not limited to statistical indicators such as p-value, statistical power, relative difference confidence interval, etc.

[0089] In a feasible embodiment, the hypothesis testing principle may be:

[0090] Hypothesis: H0: There is no statistical significance between subject group A and subject group B.

[0091] Accordingly, if Figure 5 As shown, the method of obtaining the statistically significant results between each subject group based on the hypothesis testing principle and the normal distribution model corresponding to each subject group may include:

[0092] S10131. Based on the hypothesis testing principle and the normal distribution model corresponding to each object group, calculate the probability that there is no statistical significance between each object group.

[0093] S10133. Determine whether the probability that there is no statistical significance between each object group is less than a second threshold.

[0094] S10135. If the probability that there is no statistical significance between each object group is less than the second threshold, then the hypothesis testing principle does not hold and there is statistical significance between each object group.

[0095] S10137. If the probability that there is no statistical significance between each object group is greater than or equal to the second threshold, then the hypothesis testing principle is established and there is no statistical significance between each object group.

[0096] In a feasible embodiment, the P value between the subject group A and the subject group B can be calculated according to the hypothesis test principle and the mean and variance corresponding to the normal distribution model. If the P value is less than the second threshold, it indicates that the possibility of the above hypothesis is small, and the above hypothesis is not established, which proves that there is statistical significance between the subject group A and the subject group B. Otherwise, there is no statistical significance.

[0097] S1015. According to the statistical significance results between each object group, a target information operation strategy is determined from the preset information operation strategies corresponding to each of the multiple object groups.

[0098] In the embodiments of this specification, a conclusion can be drawn as to which strategy is better between the strategy corresponding to the subject group A and the strategy corresponding to the subject group B based on statistical indicators such as p-value, statistical power, relative difference confidence interval, etc.

[0099] In a feasible embodiment, the priority of the preset information operation strategy corresponding to each object group can be ranked according to statistical indicators such as p-value, statistical power, relative difference confidence interval, etc., and the preset information operation strategy with the highest priority can be determined as the best and most suitable information operation strategy for the user.

[0100] Specifically, determining the target information operation strategy from preset information operation strategies corresponding to each of the plurality of object groups according to the statistical significance results between each object group may include:

[0101] S10151. Prioritize the preset information operation strategies corresponding to each object group according to the statistical significance results between each object group;

[0102] S10153. Use the preset information operation strategy with the highest ranking as the target information operation strategy.

[0103] like Figure 6 As shown, taking the application scenario of "determining an information operation strategy that can improve the click-through rate of a video" as an example, the information operation strategy determination method provided in the embodiments of this specification is described:

[0104] 1. Divide the users into two object groups, A and B. The preset information operation strategy corresponding to object group A is "the background of the button clicked on the video is red", and the preset information operation strategy corresponding to object group B is "the background of the button clicked on the video is green". Collect the behavior data of users in the two object groups A and B clicking on videos.

[0105] 2. For each object group, when calculating the experimental indicator data, the user behavior data can be divided into N buckets, and the related user data will be divided into the same bucket, for example, all the data of the same user will be divided into the same bucket. This bucket is the operation data set in S105.

[0106] 3. Aggregate the data in each of the N buckets and treat them as new user behavior data. Figure 6 Bucket 1 contains the behavior data of two users, user 1 and user 5. Assuming that the metric to be calculated is the video click-through rate, the aggregated new user data is <new user 1, 4, 3>, where 4 means there are four videos and 3 means there are three click behaviors in total.

[0107] 4. Treat N buckets as new raw user behavior data and use Jackknife resampling to calculate the mean and variance of the data. Assuming there are three buckets, each bucket is aggregated into new user data, as shown below. The video click-through rate indicator can be calculated based on these data.

[0108] Bucket 1: <new users 1, 4, 3>

[0109] Bucket 2: <new user 2,4,2>

[0110] Bucket 3: <new user 3,3,2>

[0111] The calculation formula for video click-through rate can be: video click-through rate = total number of video clicks / number of videos. Each time resampling is performed, the data of one bucket will be removed to obtain the video click-through rate of the remaining bucket. For example, first remove the data of bucket 1 and obtain the video click-through rate click_rate_1 as (2+2) / (4+3)=0.5714, then remove the data of bucket 2 and obtain the video click-through rate click_rate_2=(3+2) / (4+3)=0.7143, remove the data of bucket 3 and obtain the video click-through rate click_rate_3=(3+2) / (4+4)=0.625. Then you can use the three data obtained by resampling, click_rate_1, click_rate_2, click_rate_3 to calculate the mean and variance of the video click-through rate you want to estimate:

[0112] The mean value is: click_rate_mean=(click_rate_1+click_rate_2+click_rate_3) / 3,

[0113] The variance is: click_rate_var = sum(click_rate_i - click_rate_mean)^2

[0114] 5. Then treat the N buckets as new original user behavior data. At this time, since the related behavior data have been divided into the same bucket, the data between buckets are not related to each other, that is, the buckets are independent and identically distributed. Therefore, the central limit theorem can be correctly run at this time to establish a normal distribution model for these N buckets. The establishment of a normal distribution model requires the mean and variance calculated in step 4 above. Finally, according to the hypothesis testing principle, calculate the statistical data of the two object groups A and B, and obtain the statistical differences between the preset information operation strategies corresponding to object group A and the preset information operation strategies corresponding to object group B, such as calculating statistical indicators such as p-value, statistical power, and relative difference confidence interval.

[0115] 6. Finally, based on these statistical indicators, we can draw a conclusion as to which preset information operation strategy corresponding to object groups A and B is better.

[0116] The information operation strategy determination method provided in the embodiment of this specification divides the behavior data with mutual relationship into the same bucket, calculates the statistical difference between groups A and B in the AB test by the bucket Jackknife method, solves the problem of variance underestimation caused by the non-independence between the data, and can significantly reduce the error rate of the experimental conclusion, avoiding the best information operation strategy obtained by the experiment, but the best information operation strategy does not achieve the expected effect. In the 10,000 simulated AB experimental tests conducted, the error rate of the theoretical experimental conclusion should be [4.58%, 5.43%], and the error rate of the experimental conclusion obtained by the actual test is 5.3%, which is within the theoretical expected range. The error rate of the AB experimental conclusion without the bucket Jackknife correction is 19%. It can be seen that the information operation strategy determination method provided in the embodiment of this specification can significantly reduce the error rate of the experimental conclusion to the theoretical expected range, and ensure that the target information operation strategy can achieve the expected effect after it is launched (for example, in the video click scenario, the use of the target information operation strategy can significantly improve the user experience, thereby improving the video click rate).

[0117] like Figure 7 As shown, an embodiment of the present invention provides an information operation strategy determination device, which may include:

[0118] The execution result acquisition module 201 can be used to obtain the execution results of multiple object groups executing their corresponding preset information operation strategies.

[0119] The operation data determination module 203 may be configured to obtain at least one operation data corresponding to each object group based on the execution result.

[0120] The operation data set determination module 205 may be used to classify at least one operation data corresponding to each object group based on the mutual relationship between the operation data to obtain at least one operation data set corresponding to each object group.

[0121] In the embodiment of this specification, the operation data set determination module 205 may include:

[0122] The identification information acquisition unit may be used to acquire identification information of the object to which each operation data corresponding to each object group belongs.

[0123] The operation data aggregation unit may be used to aggregate the operation data of the objects in each object group that have the same identification information, to obtain at least one operation data set corresponding to each object group.

[0124] The summary information determination module 207 may be used to summarize each operation data in each operation data set corresponding to each object group to obtain summary information of each operation data set corresponding to each object group.

[0125] In the embodiment of this specification, the summary information determination module 207 may include:

[0126] The object merging unit may be used to merge the objects to which each operation data in each operation data set corresponding to each object group belongs, so as to obtain the merged objects corresponding to each operation data set in each object group.

[0127] The operation data accumulation unit may be used to accumulate the operation data of the merged objects corresponding to each operation data set in each object group to obtain summary information corresponding to each operation data set in each object group.

[0128] The resampling module 209 may be used to perform a resampling process with replacement on the summary information of each operation data set corresponding to each object group to obtain a resampling result corresponding to each object group.

[0129] In the embodiment of this specification, the resampling module 209 may include:

[0130] The summary information of each operation data set corresponding to each object group is resampled N times to obtain N resampling sets corresponding to each object group; wherein N is the number of the summary information, each resampling set includes N-1 summary information, and the i-th resampling set is a set that does not include the i-th summary information, 1≤i≤N, and i and N are both positive integers.

[0131] The establishing module 2011 can be used to establish a normal distribution model corresponding to each object group based on the resampling result corresponding to each object group.

[0132] In the embodiment of this specification, the establishment module 2011 may include:

[0133] The mean and variance determination unit may determine the mean and variance of the information operation corresponding to each object group according to the summary information in the N resampling sets corresponding to each object group.

[0134] The normal distribution model determination unit can be used to establish a normal distribution model corresponding to each object group according to the mean and variance of the information operation corresponding to each object group.

[0135] The statistical significance result determination module 2013 can be used to obtain the statistical significance result between each object group according to the hypothesis testing principle and the normal distribution model corresponding to each object group.

[0136] In the embodiment of this specification, when the hypothesis testing principle includes assuming that there is no statistical significance between each object group, the statistical significance result determination module 2013 may include:

[0137] The statistical significance probability calculation unit can be used to calculate the probability that there is no statistical significance between each object group according to the hypothesis testing principle and the normal distribution model corresponding to each object group.

[0138] The probability judgment unit can be used to judge whether the probability that there is no statistical significance between each object group is less than a second threshold.

[0139] The statistical significance determination unit can be used to determine that if the probability that there is no statistical significance between each object group is less than the second threshold, the hypothesis testing principle does not hold and there is statistical significance between each object group; if the probability that there is no statistical significance between each object group is greater than or equal to the second threshold, the hypothesis testing principle holds and there is no statistical significance between each object group.

[0140] The target information operation strategy determination module 2015 can be used to determine the target information operation strategy from the preset information operation strategies corresponding to the multiple object groups according to the statistical significance results between each object group.

[0141] In the embodiment of this specification, the target information operation strategy determination module 2015 may include:

[0142] The sorting unit can be used to prioritize the preset information operation strategies corresponding to the multiple object groups according to the statistical significance results between each object group.

[0143] The preset information operation strategy determining unit with the highest ranking may be configured to use the preset information operation strategy with the highest ranking as the target information operation strategy.

[0144] In the embodiment of this specification, the device may further include:

[0145] The operation data set quantity determination module may be used to determine whether there is an object group in which the number of operation data sets is greater than a first threshold;

[0146] The operation data set merging unit can be used to merge any number of operation data sets in the object group if there is an object group whose number of operation data sets is greater than the first threshold, to obtain a merged operation data set corresponding to the object group; wherein the merged operation data set corresponds to at least one target object.

[0147] It should be noted that the device embodiments provided in the embodiments of this specification and the above method embodiments are based on the same inventive concept.

[0148] An embodiment of the present invention also provides a device for determining an information operation strategy, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the information operation strategy determination method provided in the above method embodiment.

[0149] An embodiment of the present invention also provides a storage medium, which can be set in a terminal to store at least one instruction, at least one program, code set or instruction set related to an information operation strategy determination method in a method embodiment. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the information operation strategy determination method provided by the above-mentioned method embodiment.

[0150] Optionally, in the embodiment of this specification, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the above storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0151] The memory described in the embodiments of this specification can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, applications required for functions, etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0152] The information operation strategy determination method provided in the embodiment of the present invention can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Taking running on a server as an example, Figure 8 FIG. 1 is a hardware structure diagram of a server of an information operation strategy determination method provided by an embodiment of the present invention. Figure 8As shown, the server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 310 (the processor 310 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 330 for storing data, and one or more storage media 320 (such as one or more mass storage devices) for storing application programs 323 or data 322. Among them, the memory 330 and the storage medium 320 can be short-term storage or permanent storage. The program stored in the storage medium 320 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 310 can be configured to communicate with the storage medium 320 and execute a series of instruction operations in the storage medium 320 on the server 300. The server 300 may also include one or more power supplies 360, one or more wired or wireless network interfaces 350, one or more input and output interfaces 340, and / or one or more operating systems 321, such as Windows ServerTM, Mac OSXTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0153] The input / output interface 340 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 300. In one example, the input / output interface 340 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 340 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0154] It can be understood by those skilled in the art that Figure 8 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 8 More or fewer components as shown, or with Figure 8 Different configurations are shown.

[0155] It should be noted that the sequence of the embodiments of the present invention described above is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0156] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0157] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0158] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining an information operation strategy, characterized in that: The method comprises: Obtaining the execution results of the preset information operation strategies corresponding to the multiple object groups; Based on the execution result, obtaining at least one operation data corresponding to each object group; Classifying at least one operation data corresponding to each object group based on the mutual relationship between the operation data to obtain at least one operation data set corresponding to each object group; Summarize each operation data in each operation data set corresponding to each object group to obtain summary information of each operation data set corresponding to each object group; Performing a resampling process with replacement on the summary information of each operation data set corresponding to each object group to obtain a resampling result corresponding to each object group; Based on the resampling results corresponding to each object group, a normal distribution model corresponding to each object group is established; According to the hypothesis testing principle and the normal distribution model corresponding to each object group, a statistically significant result between each object group is obtained; when the hypothesis testing principle includes assuming that there is no statistical significance between each object group, the method of obtaining the statistically significant result between each object group according to the hypothesis testing principle and the normal distribution model corresponding to each object group includes: calculating the probability that there is no statistical significance between each object group according to the hypothesis testing principle and the normal distribution model corresponding to each object group; judging whether the probability that there is no statistical significance between each object group is less than a second threshold; if the probability that there is no statistical significance between each object group is less than the second threshold, the hypothesis testing principle is not established, and there is statistical significance between each object group; if the probability that there is no statistical significance between each object group is greater than or equal to the second threshold, the hypothesis testing principle is established, and there is no statistical significance between each object group; According to the statistical significance results between each object group, a target information operation strategy is determined from preset information operation strategies corresponding to each of the multiple object groups.

2. The method according to claim 1, characterized in that: The step of classifying at least one operation data corresponding to each object group based on the mutual relationship between the operation data to obtain at least one operation data set corresponding to each object group includes: Obtain identification information of the object to which each operation data corresponding to each object group belongs; The operation data of the objects in each object group that have the same identification information are aggregated to obtain at least one operation data set corresponding to each object group.

3. The method according to claim 2, characterized in that After aggregating the operation data of the objects with the same identification information in each object group to obtain at least one operation data set corresponding to each object group, the method further includes: Determine whether there is an object group whose number of operation data sets is greater than a first threshold; If there is an object group whose number of operation data sets is greater than the first threshold, merging any number of operation data sets in the object group to obtain a merged operation data set corresponding to the object group; The merged operation data set corresponds to at least one object.

4. The method according to claim 1, characterized in that: The step of aggregating each operation data in each operation data set corresponding to each object group to obtain summary information of each operation data set corresponding to each object group includes: Merging the objects to which each operation data in each operation data set corresponding to each object group belongs to, to obtain the merged objects corresponding to each operation data set in each object group; The operation data of the merged objects corresponding to each operation data set in each object group are accumulated to obtain summary information corresponding to each operation data set in each object group.

5. The method according to claim 1, characterized in that The step of performing a resampling process with replacement on the summary information of each operation data set corresponding to each object group to obtain a resampling result corresponding to each object group includes: Perform N resampling processes on the summary information of each operation data set corresponding to each object group to obtain N resampling sets corresponding to each object group; Wherein, N is the number of the summary information, each resampling set includes N-1 summary information, and the i-th resampling set is a set that does not include the i-th summary information, 1≤i≤N, i and N are both positive integers; Accordingly, the normal distribution model corresponding to each object group is established based on the resampling result corresponding to each object group, including: Determine the mean and variance of the information operation corresponding to each object group based on the summary information in the N resampled sets corresponding to each object group; According to the mean and variance of the information operation corresponding to each object group, a normal distribution model corresponding to each object group is established.

6. The method according to claim 1, wherein the step of determining the target information operation strategy from preset information operation strategies corresponding to each of the plurality of object groups according to the statistical significance results between each object group comprises: Prioritize the preset information operation strategies corresponding to the multiple object groups according to the statistical significance results between each object group; The preset information operation strategy with the highest ranking is used as the target information operation strategy.

7. An information operation strategy determination device, characterized in that: The device comprises: An execution result acquisition module is used to obtain the execution results of the preset information operation strategies corresponding to the execution of the multiple object groups; an operation data determination module, configured to obtain at least one operation data corresponding to each object group based on the execution result; an operation data set determination module, configured to classify at least one operation data corresponding to each object group based on the mutual relationship between the operation data, and obtain at least one operation data set corresponding to each object group; A summary information determination module, used to summarize each operation data in each operation data set corresponding to each object group to obtain summary information of each operation data set corresponding to each object group; A resampling module is used to perform a resampling process with replacement on the summary information of each operation data set corresponding to each object group to obtain a resampling result corresponding to each object group; Establishing a module, for establishing a normal distribution model corresponding to each object group based on the resampling result corresponding to each object group; a statistical significance result determination module, for obtaining the statistical significance result between each object group according to the hypothesis test principle and the normal distribution model corresponding to each object group; when the hypothesis test principle includes assuming that there is no statistical significance between each object group, the statistical significance result determination module includes: a statistical significance probability calculation unit, for calculating the probability that there is no statistical significance between each object group according to the hypothesis test principle and the normal distribution model corresponding to each object group; a probability judgment unit, for judging whether the probability that there is no statistical significance between each object group is less than a second threshold; a statistical significance determination unit, for determining that if the probability that there is no statistical significance between each object group is less than the second threshold, the hypothesis test principle is not established, and there is statistical significance between each object group; if the probability that there is no statistical significance between each object group is greater than or equal to the second threshold, the hypothesis test principle is established, and there is no statistical significance between each object group; The target information operation strategy determination module is used to determine the target information operation strategy from the preset information operation strategies corresponding to each of the multiple object groups according to the statistical significance results between each object group.

8. The device according to claim 7, characterized in that The operation data set determination module includes: An identification information acquisition unit, used to acquire identification information of an object to which each operation data corresponding to each object group belongs; The operation data summarizing unit is used to summarize the operation data of the objects in each object group with the same identification information to obtain at least one operation data set corresponding to each object group.

9. The device according to claim 8, characterized in that The device also includes: An operation data set quantity determination module, used to determine whether there is an object group in which the number of operation data sets is greater than a first threshold; an operation data set merging unit, configured to merge any number of operation data sets in the object group if there is an object group whose number of operation data sets is greater than the first threshold, to obtain a merged operation data set corresponding to the object group; wherein the merged operation data set corresponds to at least one object.

10. The device according to claim 7, characterized in that The summary information determination module includes: An object merging unit, used to merge objects belonging to each operation data in each operation data set corresponding to each object group, to obtain merged objects corresponding to each operation data set in each object group; The operation data accumulation unit is used to accumulate the operation data of the merged objects corresponding to each operation data set in each object group to obtain summary information corresponding to each operation data set in each object group.

11. The device according to claim 7, characterized in that The resampling module is used to perform N resampling processes on the summary information of each operation data set corresponding to each object group to obtain N resampling sets corresponding to each object group; wherein N is the number of the summary information, each resampling set includes N-1 summary information, and the i-th resampling set is a set that does not include the i-th summary information, 1≤i≤N, and i and N are both positive integers; Accordingly, the establishment module includes: A mean and variance determination unit, used to determine the mean and variance of the information operation corresponding to each object group according to the summary information in the N resampling sets corresponding to each object group; The normal distribution model determination unit is used to establish a normal distribution model corresponding to each object group according to the mean and variance of the information operation corresponding to each object group.

12. The device according to claim 7, characterized in that The target information operation strategy determination module includes: A sorting unit, used to prioritize the preset information operation strategies corresponding to each of the multiple object groups according to the statistical significance results between each object group; The preset information operation strategy determining unit with the highest ranking is used to use the preset information operation strategy with the highest ranking as the target information operation strategy.

13. A device for determining information operation strategy, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the information operation strategy determination method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the information operation strategy determination method as described in any one of claims 1-6.

Images