Content recommendation method and device based on AB test

Abstract

The invention relates to a content recommendation method and device based on an AB test. The method comprises the following steps: an initialization step: setting an initial overall saliency level of contents, and calculating an initial correlation matrix among the plurality of contents according to historical data of content experiment data and an initial layering weight allocated to each test combination formed by different contents and different user layers; a dynamic updating step: according to the received real-time data of the content experiment data and the historical data of the content experiment data, calculating a dynamic correlation matrix among the plurality of contents and dynamic layering weights distributed for each test combination formed by different contents and different user layers; a significance level calculation step: calculating the dynamic significance level of each test combination based on the dynamic hierarchical weight; and a statistical test step: obtaining a significance judgment result of each dynamic significance level judgment test combination based on the significance level calculation step.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a content recommendation method based on AB testing and a content recommendation device based on AB testing. BACKGROUND

[0002] With the continuous improvement of the functions and business scenarios of the recommendation system, the situations faced by the platform when launching advertisements are gradually diversified, including the number and difference of advertisement creatives, user group characteristics, time period effects, and multiple factors. In traditional recommendation methods, a single correction factor is generally used to correct the significance level of multiple hypothesis tests, which can lead to two problems: the significance level is set too conservatively, reducing the detection efficiency, or it is too loose, increasing the risk of false positives.

[0003] On the other hand, existing technical solutions mostly use fixed significance correction factors to handle the significance problem of multiple hypothesis testing in A / B testing, mainly including the following categories:

[0004] (1) FWER (family-wise error rate) control method: including Bonferroni correction, Sidak correction, etc., which strictly controls the false positive risk by reducing the significance level, suitable for independent tests, but these methods are often too conservative, significantly reducing the test efficiency, and easily leading to "insufficient significant results".

[0005] (2) FDR (false discovery rate) control method: such as Benjamini-Hochberg correction (BH), Benjamini-Yekutieli correction (BY), etc., allowing a certain proportion of false positive risk, suitable for scenarios with a large number of hypothesis tests, and can improve efficiency, but these methods mostly assume that hypothesis tests are independent or positively correlated, while in advertisement testing, different advertisement creatives and user stratification often have complex negative correlations and time dependencies, leading to inaccurate correction factor settings and affecting the accuracy of recommendation decisions.

[0006] (3) Dynamic weight allocation multiple correction method: some optimization methods start to try to adjust the correction factor by dynamically allocating weights, but they are mostly designed for specific single situations and are difficult to be systematically applied to multiple situation scenarios in the advertisement recommendation system, especially in time variation and user characteristic differences.

[0007] (4) Multiple correction prediction based on machine learning: using machine learning algorithms to predict the hypothesis test distribution of data sets, designing adaptive verification methods, and learning the error control requirements of different data sets through training models, which can accurately calculate the correction factor, but this method is relatively complex and requires high device performance, which is not suitable for scenarios that require fast decision response.

[0008] In summary, the prior art also has the technical problems of insufficient adaptability, insufficient detection efficacy, and difficulty in dynamic adjustment. SUMMARY

[0009] To solve the above problems in the prior art, the present application provides an AB test-based content recommendation method and device for achieving better content selection and recommendation through dynamic adjustment and multi-dimensional adaptive correction factors.

[0010] The AB test-based content recommendation method of one aspect of the present application is used to recommend one or more contents from a plurality of contents to one or more layers in a multi-layer user hierarchy, and the method comprises the following steps:

[0011] An initialization significance step sets an initial overall significance level of the content and calculates an initial correlation matrix between each content of the plurality of contents and an initial layering weight assigned to each test combination composed of different contents and different user hierarchies according to historical data of content experiment data;

[0012] A matrix and weight dynamic updating step calculates a dynamic correlation matrix between each content of the plurality of contents and a dynamic layering weight assigned to each test combination composed of different contents and different user hierarchies according to real-time data of the received content experiment data and historical data of the content experiment data;

[0013] A significance level calculation step calculates a dynamic significance level of each test combination based on the dynamic layering weight; and

[0014] A statistical test step judges the significance of each test combination based on the dynamic significance level obtained by the significance level calculation step.

[0015] Optionally, the statistical test step is further followed by:

[0016] A content recommendation step performs content recommendation based on the significance judgment result obtained by the statistical test step.

[0017] Optionally, the content experiment data comprises any one of the following:

[0018] The click rate of the content, the display amount of the content, the amount of content acquisition, the conversion rate of the content, the click amount of the content, and the amount of content cancellation.

[0019] Optionally, the initialization significance step comprises:

[0020] The initial overall significance level is set to 0.05;

[0021] Based on historical data from the content experiment, the initial correlation matrix between each piece of content was calculated using Pearson correlation coefficient or cosine similarity; and

[0022] The initial stratification weights are calculated based on the priority of each content in each user stratum and the historical effects of historical data based on the experimental data of the content.

[0023] Optionally, based on historical data from the content experiment, an initial correlation matrix between the content items is calculated using the Pearson correlation coefficient, including:

[0024] The initial correlation matrix ρ0 is calculated according to the following formula:

[0025]

[0026] Wherein, the numerator represents the covariance of the click-through rates of content i and content j, and the denominator represents the product of the standard deviations of the click-through rates of content i and content j.

[0027] Optionally, calculating the initial stratification weight based on the priority and historical performance of each content in each user stratum includes:

[0028] The initial segmentation weight of advertisement i in user segmentation U is calculated based on the following formula.

[0029]

[0030] Where, ρ i,j Let λ be the initial correlation matrix. i CTR is the base weighting coefficient for content i. j,U This represents the click-through rate of content j within user segment U.

[0031] Optionally, for the initial hierarchical weights The normalized weight w of advertisement i in user segment U is further obtained by normalization according to the following formula. i,U :

[0032]

[0033] Optionally, the matrix and weight dynamic update step includes:

[0034] Calculate the dynamic correlation matrix between the various contents based on the real-time data and historical data of the received content experiment data; and

[0035] Calculate the dynamic stratification weights for each test combination based on the dynamic correlation matrix and the priority of user stratification:

[0036] Optionally, calculating the dynamic correlation matrix between the various contents based on the received real-time data and historical data of the content experiment data includes:

[0037] The dynamic correlation matrix is ​​calculated based on the Pearson correlation coefficient using real-time data and historical data of the content experiment data.

[0038] Optionally, calculating the weights of each dynamic stratification based on the dynamic correlation matrix and the importance strategy of user stratification includes:

[0039] The dynamic hierarchical weight w is calculated according to the following formula. i,t,U :

[0040]

[0041] Where i represents the ad ID for which weight is to be calculated, j represents other ad IDs different from ad i, k represents the index of the indexed ad ID, and λ i For the priority of content i, CTR j This represents the click-through rate of content j.

[0042] Optionally, in the saliency level calculation step, the dynamic saliency level is calculated based on the dynamic stratification weights and a two-dimensional decreasing function of time and user grouping.

[0043] Optionally, in the significance level calculation step, the dynamic significance level α is calculated based on the following formula. i,t,U :

[0044]

[0045] Where α is the initial overall significance level, w i,t,U For dynamic hierarchical weights, e -d(t,U) A two-dimensional decreasing function grouped by time and user.

[0046] Optionally, in the statistical test step, a p-value is calculated based on the dynamic significance level calculated in the significance level calculation step, and the significance judgment result is obtained by comparing the p-value with the dynamic significance level.

[0047] Optionally, the following may be included after the recommended step:

[0048] The data collection step involves collecting real-time data of the content experiment data generated based on the content recommendations obtained in the content recommendation step; and

[0049] The real-time feedback and parameter correction step feeds back the real-time experimental data collected in the data collection step to the matrix and weight matrix and weight dynamic update step.

[0050] An A / B testing-based content recommendation device according to one aspect of the present invention is used to recommend one or more pieces of content from a plurality of content to one or more layers of a multi-level user hierarchy, the device comprising:

[0051] The initial saliency module is used to set the initial overall saliency level of the content and calculate the initial correlation matrix between the multiple contents and the initial stratification weights assigned to each test combination composed of different contents and different user stratifications based on the historical data of the content experiment data.

[0052] The matrix and weight dynamic update module is used to calculate the dynamic correlation matrix between the contents of the multiple contents and the dynamic stratification weights assigned to each test combination composed of different contents and different user stratifications based on the real-time data of the received content experiment data and the historical data of the content experiment data.

[0053] A significance level dynamic update module is used to calculate the dynamic significance level of each test combination based on the dynamic stratified weights; and

[0054] The statistical testing module is used to obtain the significance judgment results of each test combination based on the dynamic significance level update module.

[0055] Optionally, it further includes:

[0056] The content recommendation module is used to recommend content based on the significance judgment results obtained from the statistical test module.

[0057] Optionally, it further includes:

[0058] The data collection module is used to collect real-time data of content experiment data generated based on content recommendations obtained from the content recommendation module; and

[0059] The real-time feedback and parameter correction module is used to feed back the real-time experimental data collected by the data collection module to the matrix and weight dynamic update module.

[0060] Optionally, the initial saliency module sets the initial overall saliency level to 0.05, and calculates the initial correlation matrix between each content using Pearson correlation coefficient or cosine similarity based on historical data of the content experiment data, and calculates the initial stratification weight according to the priority of each content in each user stratum and the historical effect based on the historical data of the content experiment data.

[0061] Optionally, the matrix and weight dynamic update module calculates the dynamic correlation matrix between the contents of the multiple contents based on the real-time data and historical data of the received content experiment data, and calculates the dynamic stratification weight of each test combination based on the dynamic correlation matrix and the priority of user stratification.

[0062] Optionally, the matrix and weight dynamic update module calculates the dynamic correlation matrix based on the Pearson correlation coefficient using real-time data and historical data of the content experiment data.

[0063] Optionally, the saliency level dynamic update module calculates the dynamic saliency level based on the dynamic hierarchical weights and a two-dimensional decreasing function of time and user grouping.

[0064] Optionally, the statistical test module calculates a p-value based on the dynamic significance level calculated by the significance level dynamic update module and obtains the significance judgment result by comparing the p-value with the dynamic significance level.

[0065] The present invention provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned A / B testing-based content recommendation method.

[0066] A computer device according to one aspect of the present invention includes a storage module, a processor, and a computer program stored on the storage module and executable on the processor, wherein the processor executes the computer program to implement the content recommendation method based on A / B testing.

[0067] A computer program product according to one aspect of the present invention includes a computer program, which, when executed by a processor, describes a content recommendation method based on A / B testing. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating an embodiment of the content recommendation method based on A / B testing according to the present invention.

[0069] Figure 2 This is a structural block diagram of a content recommendation device based on A / B testing according to an embodiment of the present invention.

[0070] Figure 3 'a' represents a sample initial correlation matrix. Figure 3 b represents the updated correlation matrix. Figure 3 c represents the initial weight and the updated weight. Figure 3 d indicates significance.

[0071] The described and other objects and advantages of the invention will become more fully clear from the following detailed description taken in conjunction with the accompanying drawings, wherein like or similar elements are denoted by the same reference numerals. Detailed Implementation

[0072] The following are some embodiments of the present invention, intended to provide a basic understanding of the invention. They are not intended to identify key or decisive elements of the invention or to limit the scope of protection sought.

[0073] For purposes of brevity and illustrativeness, the principles of the invention are described herein primarily with reference to exemplary embodiments thereof. However, those skilled in the art will readily recognize that the same principles are equivalently applicable to all types of A / B testing-based content recommendation methods and apparatuses, and that these same principles can be implemented therein, and that any such variations do not depart from the true spirit and scope of the invention.

[0074] Furthermore, reference is made in the following description to the accompanying drawings, which illustrate specific exemplary embodiments. Electrical, mechanical, logical, and structural modifications may be made to these embodiments without departing from the spirit and scope of the invention. Moreover, while features of the invention are disclosed in conjunction with only one of several embodiments, this feature may be combined with one or more other features of other embodiments if desired and / or advantageous for any given or identifiable function. Therefore, the following description should not be considered limiting in any sense, and the scope of the invention is defined by the appended claims and their equivalents.

[0075] Terms such as “possessing” and “comprising” indicate that, in addition to having units (modules) and steps that are directly and explicitly stated in the specification and claims, the technical solution of the present invention does not exclude the presence of other units (modules) and steps that are not directly or explicitly stated.

[0076] First, some technical terms used in this invention will be briefly explained.

[0077] (1) CTR

[0078] The full Chinese name is Click-Through Rate, which refers to the ratio of the number of times an advertisement or link is clicked to the number of times it is displayed. It is used to measure the effectiveness of advertising or user interest.

[0079] (2) The significance level (usually represented by α) is the probability of incorrectly rejecting the null hypothesis when it is true. For example, α = 0.05 means there is a 5% probability of misjudging an invalid advertisement as significant.

[0080] Type I error refers to rejecting the null hypothesis when it is true (false positive error), and its probability is the significance level α. Type II error refers to not rejecting the null hypothesis when it is false (false negative), and its probability is β. The power of the test is the probability of correctly rejecting the null hypothesis, which is the probability of rejecting the null hypothesis when the alternative hypothesis is true (1–β).

[0081] (3) The binomial test is a statistical test used to test whether the probability of success in a given number of trials matches a certain probability. It is based on the binomial distribution and is applicable to trials with only two possible outcomes (such as success / failure, click / no click).

[0082] (4) The z-test is a statistical test used to test whether the sample mean is significantly different from a known population mean. It is based on the normal distribution and assumes that the sample size is large enough so that the distribution of the sample mean is approximately normal.

[0083] (5) The p-value represents the probability of observing the current sample data or more extreme data under the assumption that the null hypothesis is true. It reflects the degree of consistency between the data and the null hypothesis. The smaller the p-value, the stronger the inconsistency between the observed data and the null hypothesis, that is, the more likely it is to reject the null hypothesis.

[0084] Figure 1 This is a flowchart illustrating an embodiment of the content recommendation method based on A / B testing according to the present invention.

[0085] like Figure 1 As shown, an embodiment of the present invention provides a content recommendation method based on A / B testing, which includes the following steps:

[0086] Initialization saliency step S100: Set the initial overall saliency level of the content and calculate the initial correlation matrix between the contents of the multiple contents and the initial stratification weights assigned to each test combination composed of different contents and different user stratifications based on the historical data of the content experiment data.

[0087] Matrix and weight dynamic update step S200: Calculate the dynamic correlation matrix between the contents of the multiple contents and the dynamic stratification weights assigned to each test combination composed of different contents and different user stratifications based on the real-time data of the received content experiment data and the historical data of the content experiment data.

[0088] Significance level calculation step S300: Calculate the dynamic significance level of each test combination based on the dynamic stratified weights;

[0089] Statistical test step S400: Based on the significance level calculation step, obtain the significance judgment results of each test combination based on the dynamic significance level;

[0090] Content recommendation step S500: Based on the significance judgment result obtained from the statistical test step, content recommendation is performed;

[0091] Data collection step S600: Collect real-time data of the content experiment data generated based on the content recommendation obtained in the content recommendation step; and

[0092] Real-time feedback and parameter correction step S700: Feed back the real-time data of the experimental data collected in the data collection step to the matrix and weight matrix and weight dynamic update step S200.

[0093] The content mentioned includes, but is not limited to, advertisements, videos, and images. For ease of understanding, the following description will use advertisements as an example.

[0094] The experimental data for the content includes any one of the following: content click-through rate, content impressions, content redemption rate, content conversion rate, content clicks, and content sales volume.

[0095] The historical data of the content experiment refers to the data collected in the early or early stages of the content (e.g., advertising) experiment test to estimate initial parameters, such as key indicators like click-through rate (CTR), conversions, impressions, and clicks for different user groups and time periods.

[0096] The real-time data of the content experiment refers to the data collected in real time during the content (e.g., advertising) experiment testing process, which is used to dynamically adjust correction factors and parameters. This data also includes key indicators such as the click-through rate (CTR), conversion rate, impressions, and clicks of the advertisement, but they are continuously updated as the content is delivered.

[0097] The initialization saliency step S100 includes the following sub-steps:

[0098] The initial overall significance level was set at 0.05;

[0099] Based on historical data from the content experiment, the initial correlation matrix between each piece of content was calculated using Pearson correlation coefficient or cosine similarity; and

[0100] The initial stratification weights are calculated based on the priority of each content in each user stratum and the historical effects of historical data based on the experimental data of the content.

[0101] In the initial significance step S100, the initial correlation matrix between contents is calculated using the Pearson correlation coefficient based on historical data from the content experiment data, including:

[0102] The initial correlation matrix ρ0 is calculated according to the following formula:

[0103]

[0104] In this matrix, the numerator represents the covariance of the click-through rates of content i and content j, and the denominator represents the product of the standard deviations of the click-through rates of content i and content j. The initial relevance matrix represents "historical relevance," which is the initial static relevance and is used to provide a "benchmark reference."

[0105] In the initial saliency step S100, the initial stratification weight of advertisement i in user stratification U is calculated based on the following formula.

[0106]

[0107] Where, ρ i,j Let λ be the initial correlation matrix. i CTR is the base weighting coefficient for content i. j,U This represents the click-through rate of content j within user segment U.

[0108] In the initial saliency step S100, for the initial stratification weights The normalized weight w of advertisement i in user segment U is further obtained by normalization according to the following formula. i,U :

[0109]

[0110] The matrix and weight dynamic update step S200 includes the following sub-steps:

[0111] Calculate the dynamic correlation matrix between the various contents based on the real-time data and historical data of the received content experiment data; and

[0112] Calculate the dynamic stratification weights for each test combination based on the dynamic correlation matrix and the priority of user stratification:

[0113] In the matrix and weight dynamic update step S200, the dynamic correlation matrix is ​​calculated based on the Pearson correlation coefficient using the real-time data and historical data of the content experiment data (i.e., using the same calculation formula as the initial correlation matrix).

[0114] In the matrix and weight dynamic update step S200, the dynamic hierarchical weight w is calculated according to the following formula.i,t,U :

[0115]

[0116] Where, λ i For the priority of content i, CTR j This represents the click-through rate of content j.

[0117] Where i represents the ad ID for which weight is to be calculated, j represents other ad IDs different from ad i, k represents the index of the indexed ad ID (used for summing the normalized denominator), j≠i means that for ad i, the relevance and CTR of all ads except itself are summed, and j≠k means that the sum of the weights of all ads is ensured to be calculated based on the same relevance and CTR.

[0118] In the significance level calculation step S300, the dynamic significance level is calculated based on the dynamic stratification weights and the two-dimensional decreasing function of time and user grouping. Specifically, the dynamic significance level α is calculated based on the following formula. i,t,U :

[0119]

[0120] Where α is the initial overall significance level, w i,t,U For dynamic hierarchical weights, e -d(t,U) A two-dimensional decreasing function grouped by time and user.

[0121] In the statistical test step S400, the p-value is calculated based on the dynamic significance level calculated in the significance level calculation step, and the significance judgment result is obtained by comparing the p-value with the dynamic significance level.

[0122] Next, an embodiment of the present invention will be described, which is a content recommendation device based on A / B testing.

[0123] Figure 2 This is a schematic diagram of the structure of a content recommendation device based on A / B testing according to an embodiment of the present invention.

[0124] like Figure 2 As shown, the A / B test-based content recommendation device of one embodiment of the present invention mainly includes: an initial saliency module 100, a matrix and weight dynamic update module 200, a saliency level dynamic update module 300, a statistical test module 400, a data collection module 500, a real-time feedback and parameter correction module 600, and a recommendation module 700.

[0125] The specific functions of these modules are explained in detail below.

[0126] The initialization significance module 100 is used to set the initial overall significance level, the correlation matrix, and the initial stratification weights. Specifically, the initialization significance module 100 performs the following functions: setting an initial overall significance level α (e.g., 0.05); calculating the Pearson correlation coefficient or cosine similarity between ad creatives using historical ad content experimental data to construct the initial correlation matrix ρ0; and initializing the stratification weights w of each test unit according to business strategies such as the time period to which the ad belongs and user segmentation. i,0,U .

[0127] The matrix and weight dynamic update module 200 is used to introduce the real-time correlation matrix ρ t and dynamic hierarchical weights w i,t,U This enables the system to adjust correction factors in near real-time, capture the temporal and user characteristic correlations between advertisements, and assign independent weights to different advertisements, time periods, and user segments.

[0128] As an example, the matrix and weight dynamic update module 200 performs the following actions:

[0129] For real-time inflow of content experiment data (e.g., CTR, clicks, impressions, conversion rate, etc.) into the system, the real-time relevance matrix ρ is updated on a rolling basis according to the time window t. t A weighted moving average mechanism was added to avoid jitter;

[0130] Use optimization functions (such as mutual information or covariance sparse regularization terms) to regularize the correlation matrix to prevent collinearity interference;

[0131] Based on the new matrix ρ t The importance strategy for user groups is used to recalculate the dynamic hierarchical weights w. i,t,U As an example, the dynamic hierarchical weight w can be calculated using the following formula. i,t,U :

[0132]

[0133] Where, λ i For the priority of content i, CTR j This represents the click-through rate of content j.

[0134] Where i represents the ad ID for which weight is to be calculated, j represents other ad IDs different from ad i, k represents the index of the indexed ad ID (used for summing the normalized denominator), j≠i means that for ad i, the relevance and CTR of all ads except itself are summed, and j≠k means that the sum of the weights of all ads is ensured to be calculated based on the same relevance and CTR.

[0135] The significance level dynamic update module 300 is used to propose a hierarchical decreasing formula for setting the significance level. This allows the control strength of the correction factor to be gradually reduced during data accumulation and multiple tests in advertising effectiveness evaluation, ensuring more effective control of the significance level in long-term testing. Specifically, the dynamic significance level α is calculated as follows: i,t,U :

[0136]

[0137] Where d(t,U) represents a decreasing function of time and user group in two dimensions, and d(t,U) is set as:

[0138] d(t,U)=β1·elapsedTime(t)+β2·userEntropy(U)

[0139] The decreasing function indicates that the longer the time (e.g., the longer the advertisement runs), the stricter the significance should be; the more dispersed the user segmentation (the higher the entropy), the stricter the control of false positives should be.

[0140] The statistical testing module 400 collects content experiment data for each advertisement within each time window, such as CTR, conversion rate, and click-through rate. Historical versions for the same user group serve as a control group. The p-value is estimated using a binomial test or z-test. The p-value for each advertisement combination is compared with its dynamically adjusted significance level to determine significance. The significance calculation results are directly synchronized to the recommendation module 700. The recommendation module 700 uses the significance calculation results from the statistical testing module 400 to recommend content. For example, the recommendation module 700 identifies user groups (e.g., existing users, new users, highly active users) and matches significant combinations based on the significance calculation results for advertisement recommendations. As an example, significant advertisement versions will be prioritized, while non-significant advertisements can be delayed or retested as alternative versions.

[0141] The data collection module 500 is used to collect information such as CTR, impressions, redemptions, and sales volume of user behavior and transaction behavior (coupons, tickets, etc.) after being recommended by the recommendation module 700, and provides it to the real-time feedback and parameter correction module 600. The real-time feedback and parameter correction module 600 can update the subsequent dynamic stratification weights and significance levels based on the parameters of the real-time feedback to ensure more effective control of the significance level in long-term testing.

[0142] Next, an example of an A / B testing-based content recommendation method of the present invention will be described. In this example, the advertising test scenario constructed using the dimensions shown in Table 1 below is selected:

[0143] Table 1

[0144] advertising user groupings A new users B old users C highly active users

[0145] Table 1 shows three ads: A, B, and C, which are targeted at new users, existing users, and highly active users, respectively. The goal is to select the optimal ad mix while ensuring significant ad performance across different user groups.

[0146] An example of the A / B testing-based content recommendation method of the present invention includes the following steps:

[0147] (1) Initial data collection steps

[0148] As initial data, historical CTR data is collected as an example to estimate the initial correlation matrix ρ0. For example, the following 6-day historical data (in CTR) is collected as shown in Table 2:

[0149] Table 2

[0150] date A B C Day 1 (D1) 0.049 0.078 0.067 Day 2 (D2) 0.060 0.038 0.038 Day 3 (D2) 0.033 0.073 0.060 Day 4 (D3) 0.065 0.031 0.078 Day 5 (D4) 0.072 0.041 0.039 Day 6 (D5) 0.039 0.045 0.056

[0151] (2) Steps for calculating the initial correlation matrix ρ0

[0152] Based on the relevance of each advertisement, an initial significance level was set (α = 0.05), and the initial relevance matrix ρ0 was calculated using the Pearson correlation coefficient.

[0153]

[0154] Where the numerator represents the covariance of the click-through rates of ad i and ad j, and the denominator represents the product of the standard deviations of the click-through rates of ad i and ad j, thus the following can be calculated:

[0155]

[0156] (3) Initial hierarchical weights w i,0,U Calculation steps

[0157] Initial stratification weights are generated for each combination. These initial weights are assigned based on the ad's priority and historical performance within the user segment. The calculation formula is as follows:

[0158]

[0159] Among them, the initial hierarchical weights λ represents the original weight of ad i in user segment U, λ is the basic weight coefficient, and j≠i represents the sum of all ad relevance and CTR except for ad i itself.

[0160] Here, the formula means that the original weight of ad i in user segment U is the result of weighting its correlation with the click-through rate (CTR) of other ads j. In other words, the weight of ad i depends not only on its own CTR, but also on the influence of ads related to it.

[0161] Where λ={A:1.0,B:1.2,C:1.1} is the empirical prior, the current CTR values ​​are shown in Table 3:

[0162] Table 3

[0163] advertising new users old users highly active users A 0.053 0.044 0.054 B 0.049 0.049 0.064 C 0.058 0.051 0.046

[0164] Here, taking advertisement A targeting new users as an example, the layered weighting...

[0165]

[0166] To make the hierarchical weights comparable, it is necessary to compare all Perform normalization:

[0167]

[0168] This formula represents the normalized weight of ad i in user segment U, which is its original weight. The sum of the original weights of ad i in the same user segment The ratio of the weights of all advertisements is normalized to the range of [0,1] and the sum is 1.

[0169] The initial hierarchical weights w calculated in this example i,0,U As shown in Table 4:

[0170] Table 4

[0171] new users old users highly active users A 0.06863 0.12606 0.12786 B 0.08001 0.14236 0.13220 C 0.06421 0.11394 0.14472

[0172] (4) Correlation matrix update steps

[0173] Next, we describe the dynamic update scenario. In this example, we assume that one day of data is added: day 7 (D7).

[0174] After the experiment begins, the system will automatically update the relevance matrix at each time period (e.g., in days) to reflect the dynamic changes in advertising performance. In this example, the CTR data for the newly added 7th day is shown in Table 5:

[0175] Table 5

[0176] Day 7 (D7) new users old users highly active users A 0.055 0.058 0.068 B 0.058 0.045 0.069 C 0.041 0.042 0.064

[0177] Calculate and update the correlation matrix ρ according to the following formula. t :

[0178]

[0179] Then update the correlation matrix ρ t :

[0180]

[0181] (5) Dynamic weight w i,t,U Update steps

[0182] Based on the data from D7, and still using the following weighting formula, calculate the dynamic weights.

[0183]

[0184] For the calculated dynamic weights Normalization is performed:

[0185]

[0186] The final calculated dynamic weight w i,t,U As shown in Table 6 below:

[0187] Table 6

[0188] new users old users highly active users A 0.08147 0.13877 0.13933 B 0.07707 0.14188 0.12250 C 0.07160 0.12540 0.10199

[0189] (6) Steps for calculating dynamic significance level

[0190] Based on dynamic weight w i,t,U Calculate the significance level α i,t,U The calculation formula is as follows:

[0191]

[0192] Where α is set to 0.05, and the discount factor d t,U =0.1, so in this example α i,t,U The calculation results are shown in Table 7 below:

[0193] Table 7

[0194] new users old users highly active users A 0.003686 0.006278 0.006303 B 0.003487 0.006419 0.005542 C 0.003239 0.005673 0.004614

[0195] (8) Steps for observing p-values ​​and determining significance

[0196] By comparing historical data, the p-values ​​obtained using the z-test method are shown in Table 8 below:

[0197] Table 8

[0198] advertising user groupings p i,t,U ]]> A new users 0.038 A old users 0.026 A highly active users 0.046 B new users 0.046 B old users 0.004 B highly active users 0.027 C new users 0.050 C old users 0.022 C highly active users 0.075

[0199] Here, as a conclusion of the experiment, only advertisement B is significantly effective for existing users because:

[0200] p B,老用户 =0.004<α B,老用户 =0.006419

[0201] In other words, the calculated p-value is less than the pre-set significance level. Therefore, we reject the null hypothesis and conclude that the CTR of Ad B targeting existing users is significantly higher than that of the control group or other ad versions. Thus, we can conclude that only Ad B targeting existing users is significant. This is because, in this specific context, the calculated p-value is small enough, less than the pre-set significance level, to provide sufficient statistical evidence to reject the null hypothesis and conclude that there is a significant difference in CTR between Ad B and existing users. This conclusion is of great significance for ad optimization because it indicates which ad version performs best in which user segment, thus providing advertisers with targeted optimization suggestions.

[0202] Next, refer to Figure 3 (a) to (d) illustrate the correlation matrix and significance in this example. Figure 3 (a) represents the example initial correlation matrix. Figure 3 (b) represents the updated correlation matrix. Figure 3 (c) represents the initial weights and the updated weights. Figure 3 (d) indicates significance.

[0203] like Figure 3 (a) and Figure 3 As shown in b, these two matrices demonstrate the correlation between variables A, B, and C (A represents new users, B represents old users, and C represents highly active users). Each cell in the matrix represents the correlation value between two variables, with values ​​ranging from -1 to 1. Figure 3 The bar chart in c compares the initial weights with the updated weights. The horizontal axis represents different combinations of variables (such as A_new user, B_new user, etc.), and the vertical axis represents the weight values. Figure 3 In cell d, significance is displayed using 0 and 1, where a value of 1 indicates that the corresponding relationship is significant, and a value of 0 indicates that it is not significant. Figure 3 d indicates that it is significant that advertisement B targets existing users.

[0204] Thus, during the testing process of the A / B testing-based content recommendation device of this invention, the stratification weights and relevance matrix are continuously optimized based on feedback data, thereby continuously improving the saliency detection effect and correction accuracy in multi-scenario A / B testing. For example, if the p-value of advertisement A in the new user stratification remains close to the saliency level, the system will increase the stratification weight of advertisement A and adjust the relevance of advertisement A to other advertisements in the relevance matrix to enhance the detection sensitivity of advertisement A. Therefore, through the stratified adaptive dynamic correction factor method, the A / B testing-based content recommendation device of this invention achieves flexible saliency level control and optimized advertisement recommendation in advertisement delivery, effectively improving the accuracy of advertisement selection and user experience.

[0205] As described above, the hierarchical adaptive dynamic correction factor scheme proposed by the A / B testing-based content recommendation device and the A / B testing-based content recommendation method of the present invention can adapt to different situational changes in advertising in real time by dynamically adjusting the relevance matrix and hierarchical weights, thereby improving the significance level control accuracy and system detection efficiency of multiple hypothesis testing. Specifically, it can achieve the following technical effects:

[0206] By utilizing a dynamic correlation matrix and a hierarchical weight allocation mechanism, it can adapt to various contextual changes such as multiple content ideas, different user groups, and time periods, thereby enabling precise control of the significance level.

[0207] By using a hierarchical adaptive correction factor allocation method, we can maintain high test power, improve the significance detection rate, reduce false positive rate and redundant recommendations, thereby improving test power.

[0208] Based on real-time feedback, the system dynamically adjusts during multiple hypothesis testing to adapt to differences in content relevance, time dependence, and user preferences in content recommendation devices, thereby flexibly adapting to the dynamic nature of the content environment.

[0209] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art can conceive of other feasible variations or substitutions based on the technical scope disclosed in the present invention, and such variations or substitutions are all covered within the scope of protection of the present invention. Where there is no conflict, the embodiments of the present invention and the features thereof can also be combined with each other. The scope of protection of the present invention is determined by the claims.

Claims

1. A content recommendation method based on A / B testing, wherein the content recommendation method is used to recommend one or more pieces of content from multiple content sets to one or more layers of a multi-level user hierarchy, characterized in that, The method comprises the following steps: an initialization significance step of setting an initial overall significance level of the contents and calculating an initial correlation matrix between each content of the plurality of contents and an initial stratification weight assigned to each test combination stratified by different contents and different user stratifications according to historical data of the content experiment data; a matrix and weight dynamic updating step of calculating a dynamic correlation matrix between each content of the plurality of contents and a dynamic stratification weight assigned to each test combination stratified by different contents and different user stratifications according to real-time data of the received content experiment data and the historical data of the content experiment data; a significance level calculation step of calculating a dynamic significance level of each test combination based on the dynamic stratification weight; and a statistical test step of judging the significance of each test combination based on the dynamic significance level obtained by the significance level calculation step. 2.The AB test based content recommendation method of claim 1, wherein, Further comprising the following steps after the statistical test step: a content recommendation step of making content recommendation based on the significance judgment result obtained by the statistical test step.

3. The AB test-based content recommendation method of claim 2, wherein the content experiment data comprises any one of the following: a click rate of the content, a display quantity of the content, a collection quantity of the content, a conversion rate of the content, a click quantity of the content, and a cancellation quantity of the content. 4.The AB test based content recommendation method of claim 3, wherein, the initialization significance step comprises: setting the initial overall significance level as 0.05; calculating the initial correlation matrix between each content based on the historical data of the content experiment data by using a Pearson correlation coefficient or a cosine similarity; and calculating the initial stratification weight according to a priority of each content in each user stratification and a historical effect based on the historical data of the content experiment data.

5. The AB test-based content recommendation method of claim 4, wherein calculating the initial correlation matrix between each content based on the historical data of the content experiment data by using a Pearson correlation coefficient comprises: calculating the initial correlation matrix ρ0 according to the following formula: wherein the numerator represents a covariance of the click rates of the content i and the content j, and the denominator represents a product of standard deviations of the click rates of the content i and the content j.

6. The AB test-based content recommendation method of claim 5, wherein the calculating the initial stratification weight according to the priority of each content in each user stratification and the historical effect comprises: The initial stratification weight of an advertisement i in a user stratum U is calculated based on the following formula wherein ρ i,j is the initial correlation matrix, λ i is the base weight coefficient of content i, CTR j,U denotes the click rate of content j in user stratum U.

7. The AB test-based content recommendation method of claim 6, wherein for the initial stratification weights Further, the normalized weight w of the advertisement i in the user stratification U is obtained by normalizing according to the following formula i,U :

8. The AB test-based content recommendation method of claim 7, wherein the matrix and weight dynamic updating step comprises: calculating the dynamic correlation matrix between each content of the plurality of contents according to the real-time data of the received content experiment data and the historical data of the content experiment data; and calculating the dynamic stratification weight of each test combination according to the dynamic correlation matrix and a priority of the user stratification.

9. The AB test-based content recommendation method of claim 8, wherein The calculating of the dynamic correlation matrix between each content of the plurality of contents based on the real-time data of the content experiment data and the historical data of the content experiment data comprises: The dynamic correlation matrix is calculated based on Pearson correlation coefficient using the real-time data of the content experiment data and the historical data of the content experiment data.

10. The AB test based content recommendation method of claim 9, wherein, The calculating of the dynamic stratification weight based on the dynamic correlation matrix and the importance strategy of user stratification comprises: The dynamic stratification weight w is calculated according to the following formula i,t,U : where i denotes the advertisement number currently to be calculated the weight, j denotes the other advertisement number different from the advertisement i, k denotes an index of the index advertisement number, λ i is the priority of the content i, CTR j denotes the click rate of the content j.

11. The AB test based content recommendation method of claim 10, wherein, In the significance level calculating step, the dynamic significance level is calculated based on the dynamic stratification weight and a two-dimensional decreasing function of time and user grouping.

12. The AB test based content recommendation method of claim 11, wherein, In the significance level calculation step, a dynamic significance level a is calculated based on the following equation i,t,U : where a is the initial overall significance level, w i,t,U is a dynamic stratification weight, e -d(t,U) is a two-dimensional decreasing function of time and user group.

13. The AB test based content recommendation method of claim 2, wherein, In the statistical test step, the p-value is calculated based on the dynamic significance level calculated in the significance level calculating step, and the significance judgment result is obtained according to the comparison between the p-value and the dynamic significance level. 14.The AB test based content recommendation method of claim 13, wherein, Further comprising: a data collecting step for collecting the real-time data of the content experiment data generated based on the content recommendation obtained in the recommending step; and a real-time feedback and parameter correction step for feeding back the real-time data of the content experiment data collected in the data collecting step to the matrix and weight dynamic updating step.

15. An AB test-based content recommendation apparatus for recommending one or more contents among a plurality of contents to one or more layers in a multi-layer user stratification, the apparatus comprising: comprise: an initial significance module for setting an initial overall significance level of the content and calculating an initial correlation matrix between each content of the plurality of contents and an initial stratification weight assigned to each test combination composed of different content and different user stratification based on the historical data of the content experiment data; a matrix and weight dynamic updating module for calculating a dynamic correlation matrix between each content of the plurality of contents and a dynamic stratification weight assigned to each test combination composed of different content and different user stratification based on the real-time data of the content experiment data received and the historical data of the content experiment data; a significance level dynamic updating module for calculating a dynamic significance level of each test combination based on the dynamic stratification weight; and a statistical test module for judging a significance judgment result of each test combination based on the dynamic significance level obtained by the significance level dynamic updating module. 16.The AB test based content recommendation apparatus of claim 15, wherein, Further comprising: a content recommendation module for making content recommendation based on the significance judgment result obtained by the statistical test module.

17. The AB test based content recommendation apparatus of claim 16, wherein, Further comprising: a data collecting module for collecting the real-time data of the content experiment data generated based on the content recommendation obtained by the content recommendation module; and a real-time feedback and parameter correction module for feeding back the real-time data of the content experiment data collected by the data collecting module to the matrix and weight dynamic updating module.

18. The AB test based content recommendation device of claim 17, wherein, The initialization significance module sets an initial overall significance level as 0.05, and calculates the initial correlation matrix between each content based on historical data of content experiment data by using Pearson correlation coefficient or cosine similarity, and calculates the initial stratification weight according to the priority of each content in each user stratification and historical effect based on historical data of the content experiment data. 19.The AB test based content recommendation device of claim 18, wherein, The matrix and weight dynamic updating module calculates a dynamic correlation matrix between each content of the plurality of contents according to real-time data of the received content experiment data and historical data of the content experiment data, and calculates a dynamic stratification weight of each test combination according to the dynamic correlation matrix and priority of user stratification. 20.The AB test based content recommendation device of claim 19, wherein, The matrix and weight dynamic updating module calculates the dynamic correlation matrix based on Pearson correlation coefficient by using real-time data of the content experiment data and historical data of the content experiment data. 21.The AB test based content recommendation device of claim 20, wherein, The significance level dynamic updating module calculates a dynamic significance level based on the dynamic stratification weight, and a two-dimensional decreasing function of time and user grouping. 22.The AB test based content recommendation device of claim 21, wherein, The statistical test module calculates a p-value based on the dynamic significance level calculated by the significance level dynamic updating module, and obtains the significance judgment result according to comparison between the p-value and the dynamic significance level. 23.A computer readable medium having a computer program stored thereon, wherein, The computer program is executed by a processor to implement the AB test based content recommendation method of any one of claims 1-14. 24.A computer device comprising a storage module, a processor, and a computer program stored on the storage module and executable on the processor, wherein, The processor executes the computer program to implement the AB test based content recommendation method of any one of claims 1-14. 25.A computer program product comprising a computer program, wherein, The computer program is executed by a processor to implement the AB test based content recommendation method of any one of claims 1-14.

Images