Content pushing strategy processing method and device, computer device and storage medium

Abstract

The application relates to a content pushing strategy processing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a reference content operation record set corresponding to a reference content pushing strategy and a test content operation record set corresponding to a test content pushing strategy; dividing the reference content operation record set and sampling the divided set multiple times to obtain a sampling reference record sub-set corresponding to each sampling, dividing the test content operation record set and sampling the divided set multiple times to obtain a sampling test record sub-set corresponding to each sampling; and obtaining a strategy test result corresponding to the test content pushing strategy based on the difference between a reference operation index value and a test operation index value. The above scheme quickly obtains the strategy test result corresponding to the test content pushing strategy based on the difference between the reference operation index value and the test operation index value, and effectively improves the processing efficiency of the content pushing strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network, in particular to a content pushing strategy processing method and device, computer equipment and storage medium. BACKGROUND

[0002] In the media content pushing scenario, the content is pushed according to a certain content pushing strategy. In order to obtain better pushing effect, the pushing data corresponding to the content pushing strategy is often processed before pushing, for example: the content pushing strategy with good pushing effect is determined.

[0003] In the traditional technology, the data volume to be processed for processing the pushing data corresponding to different content pushing strategies is very large, so that the processing efficiency of the content pushing strategy is relatively low.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] Therefore, it is necessary to provide a content pushing strategy processing method, device, computer equipment and storage medium for the above technical problems.

[0006] A content pushing strategy processing method, the method comprising: obtaining a reference content operation record set corresponding to a reference content pushing strategy and a test content operation record set corresponding to a test content pushing strategy; dividing the reference content operation record set to obtain a plurality of reference record sub-sets, and dividing the test content operation record set to obtain a plurality of test record sub-sets; sampling the reference record sub-sets multiple times to obtain a sampling reference record sub-set corresponding to each sampling, and sampling the test record sub-sets multiple times to obtain a sampling test record sub-set corresponding to each sampling; taking the sampling reference record sub-set as a sample calculation unit to obtain a reference operation index value corresponding to the sampling reference record sub-set; taking the sampling test record sub-set as a sample calculation unit to obtain a test operation index value corresponding to the sampling test record sub-set; and obtaining a strategy test result corresponding to the test content pushing strategy based on the difference between the reference operation index value and the test operation index value.

[0007] A content pushing strategy processing apparatus, the apparatus comprising: a record set obtaining module configured to obtain a reference content operation record set corresponding to a reference content pushing strategy and a test content operation record set corresponding to a test content pushing strategy; a record set dividing module configured to divide the reference content operation record set to obtain a plurality of reference record sub-sets and divide the test content operation record set to obtain a plurality of test record sub-sets; a set sampling module configured to sample the reference record sub-sets multiple times to obtain a sampling reference record sub-set corresponding to each sampling, and sample the test record sub-sets multiple times to obtain a sampling test record sub-set corresponding to each sampling; an index value obtaining module configured to take the sampling reference record sub-set as a sample calculation unit to obtain a reference operation index value corresponding to the sampling reference record sub-set, and take the sampling test record sub-set as a sample calculation unit to obtain a test operation index value corresponding to the sampling test record sub-set; and a test result obtaining module configured to obtain a strategy test result corresponding to the test content pushing strategy based on a difference between the reference operation index value and the test operation index value.

[0008] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program: obtaining a reference content operation record set corresponding to a reference content pushing strategy and a test content operation record set corresponding to a test content pushing strategy; dividing the reference content operation record set to obtain a plurality of reference record sub-sets and dividing the test content operation record set to obtain a plurality of test record sub-sets; sampling the reference record sub-sets multiple times to obtain a sampling reference record sub-set corresponding to each sampling, and sampling the test record sub-sets multiple times to obtain a sampling test record sub-set corresponding to each sampling; taking the sampling reference record sub-set as a sample calculation unit to obtain a reference operation index value corresponding to the sampling reference record sub-set; taking the sampling test record sub-set as a sample calculation unit to obtain a test operation index value corresponding to the sampling test record sub-set; and obtaining a strategy test result corresponding to the test content pushing strategy based on a difference between the reference operation index value and the test operation index value.

[0009] A computer readable storage medium, having stored thereon a computer program, the computer program being executed by a processor to implement the following steps: obtaining a reference content operation record set corresponding to a reference content push strategy and a test content operation record set corresponding to a test content push strategy; dividing the reference content operation record set to obtain a plurality of reference record sub-sets, and dividing the test content operation record set to obtain a plurality of test record sub-sets; sampling the reference record sub-sets multiple times to obtain a sampling reference record sub-set corresponding to each sampling, and sampling the test record sub-sets multiple times to obtain a sampling test record sub-set corresponding to each sampling; taking the sampling reference record sub-set as a sample calculation unit to obtain a reference operation index value corresponding to the sampling reference record sub-set; taking the sampling test record sub-set as a sample calculation unit to obtain a test operation index value corresponding to the sampling test record sub-set; and obtaining a strategy test result corresponding to the test content push strategy based on the difference between the reference operation index value and the test operation index value.

[0010] The content push strategy processing method, device, computer device and storage medium described above respectively divide a reference content operation record set corresponding to a reference content push strategy and a test content operation record set corresponding to a test content push strategy, and sample the record sub-sets obtained by the division multiple times. The sub-sets obtained by the sampling realize the reorganization of the content operation record set. The reference operation index value and the test operation index value corresponding to the sub-sets obtained by the sampling are obtained at the set granularity. At this time, the reference operation index value and the test operation index value are greatly reduced in data size relative to the data corresponding to the reference content operation record set and the test content operation record set, and the diversity of the sample used for evaluating the strategy can be ensured. Therefore, the test efficiency and the test accuracy of the content push strategy are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 An application environment diagram of the content push strategy processing method in an embodiment;

[0012] Figure 2 A flowchart of the content push strategy processing method in an embodiment;

[0013] Figure 3 A flowchart of the bucketing in an embodiment;

[0014] Figure 4 A flowchart of the multiple sampling in an embodiment;

[0015] Figure 5 An interface display diagram of a target terminal in an embodiment;

[0016] Figure 6This is a schematic diagram of the interface display of the target terminal in another embodiment;

[0017] Figure 7 This is a schematic diagram of the interface display of the target terminal in another embodiment;

[0018] Figure 8 This is a schematic diagram of the system structure for implementing the content push strategy processing method in one embodiment;

[0019] Figure 9 This is a flowchart illustrating the content push strategy processing method in another embodiment;

[0020] Figure 10 This is a flowchart illustrating the push strategy processing method in another embodiment;

[0021] Figure 11 This is a structural block diagram of a content push strategy processing device in one embodiment;

[0022] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] The content push strategy processing method provided in this application can be applied to, for example... Figure 1 The application environment shown includes terminal 102, server 104, and terminal 106. Terminal 104 communicates with server 104 via a network. Terminal 102 sends a query request to server 104. Upon receiving the query request, server 104 retrieves a set of reference content operation records and a set of test content operation records from the database. Based on these sets, server 104 performs segmentation and sampling processing. Based on the reference operation indicator values ​​and test operation indicator values ​​corresponding to the segmented and sampled subsets, and the difference between these values, server 104 obtains the strategy test result corresponding to the test content push strategy. Server 104 returns the strategy test result to terminal 102 and pushes the target content to terminal 106 based on the result. Terminals 102 and 106 can be, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers. Figure 1Two terminals 106 are shown in FIG. 1. In actual application scenarios, the number of terminals 106 can be more or less.

[0025] In one embodiment, as shown in FIG. 2, a content pushing strategy processing method is provided. This embodiment takes the method applied to a server as an example. The method includes the following steps: Figure 2

[0026] S202, obtaining a reference content operation record set corresponding to a reference content pushing strategy and a test content operation record set corresponding to a test content pushing strategy.

[0027] In a content pushing scenario, a server needs to push content to terminals and the like. The pushed content refers to media content used for pushing, which can be video, news, articles, advertisements, and the like. Taking advertisements as an example, the server can push advertisements to terminals according to a pushing strategy.

[0028] The pushing strategy refers to a strategy used for content pushing, which can include at least one of a pushing manner, a pushing object, or a pushing content. The pushing manner can include at least one of a pushing time, a pushing duration, or a pushing platform. The pushing object can include a user group to be pushed, which can include at least one of users of different genders, ages, or regions. The pushing content can include specific media content to be pushed, which can be different advertisement scripts. The pushing platform can include a platform used for content pushing, which can be various types of web pages or application programs.

[0029] The reference content pushing strategy and the test content pushing strategy are two strategies used for content pushing. The reference content pushing strategy refers to a strategy used for reference comparison of the test content pushing strategy. In one embodiment, the reference content pushing strategy can be a historical content pushing strategy, and the test content pushing strategy is a newly determined content pushing strategy. In the testing process of the content pushing strategy, whether the test content pushing strategy is an effective content pushing strategy can be determined based on the pushing effect difference of the test content pushing strategy relative to the reference content pushing strategy. In one embodiment, the reference content pushing strategy is a content pushing strategy used for comparison of the test content pushing strategy. Therefore, the reference content pushing strategy and the test content pushing strategy can be considered as two groups of content pushing strategies for mutual comparison, that is, the reference content pushing strategy is determined as a control group, and the test content pushing strategy is determined as an experimental group, and then the test of the experimental group is realized based on the control group to obtain a corresponding experimental test result. The experimental group and the control group can be represented as group A experiment and group B experiment, respectively. Therefore, the process of testing the test content pushing strategy can be called A / B testing.

[0030] ​In an embodiment, the reference content push strategy can be one, and the number of test content push strategies can be at least one. When the test content push strategy is multiple, the test of multiple test content push strategies can be implemented in one test, and the optimal content push strategy can be determined among the test content push strategies according to the strategy test result. In an embodiment, the reference content push strategy can be selected from multiple historical content push strategies. The test content push strategy can be selected from multiple current content push strategies. Among them, the current content push strategy can include a newly determined content push strategy.

[0031] Before the content push strategy is pushed, the user to be pushed can be acquired, and then the corresponding push content is pushed to the user according to the content push strategy. In an embodiment, the user to be pushed by the reference content push strategy is referred to as a reference user, and the user to be pushed by the test content push strategy is referred to as a test user. When the user views the push content, the user can perform at least one operation according to the user's own needs, such as at least one operation of liking, forwarding, or re-editing. At this time, the terminal used by the user can acquire the record information corresponding to the operation, and the terminal can send the content operation record information to the server. Among them, the content operation record can also be referred to as operation record. In an embodiment, the server can generate a content operation record for each operation, and when a user triggers multiple operations, the user can correspond to multiple content operation records. When processing the content operation record, the content operation records corresponding to the same user can be bundled together for processing, and then the processing of the content operation record is realized with the user as the research unit.

[0032] The reference user and the test user can be users determined from sample users, and the reference user and the test user can correspond to the same user attributes. Among them, the sample user can be a user to be determined for content push in advance, and can be a user registered in a specific application. The user attributes can include at least one of name, gender, age, preference, or geographic location. In an embodiment, the sample users can be randomly divided into two groups, and the two groups of users are uniformly distributed in the user attributes, for example: the male and female ratio in the reference user and the test user corresponds to the same, and the user ratio of each age group corresponds to the same, and the two groups of users are respectively taken as the reference user and the test user. In an embodiment, when the reference and test users are determined from the sample users, the consistency of the reference user and the test user in the user attributes needs to be ensured as much as possible, so that there is no difference in the user pushed when the strategy test is performed, and the difference is only in the different strategies adopted, and then the test content push strategy can be reliably tested based on the content operation record of the user to the push content.

[0033] In an embodiment, the reference content operation record set is a set of content operation records corresponding to the reference user, which can be obtained by arranging the content operation records in sequence, and all the arranged content operation records can be taken as the reference content operation record set. In an embodiment, the test content operation record set is a set of content operation records corresponding to the test user, which can be obtained by arranging the content operation records in sequence, and all the arranged content operation records can be taken as the test content operation record set.

[0034] In an embodiment, the reference content operation record set corresponding to the target dimension can be obtained, and the test content operation record set corresponding to the target dimension can be obtained. The target dimension is a data taking condition for the current test process, which can also be referred to as an analysis dimension. In the content pushing scenario, the target dimension can be an industry dimension, such as at least one of the automobile industry, the spare parts industry, or the cosmetic industry, etc. In an embodiment, the target dimension can also include at least one of a time dimension, a sample dimension, an experiment dimension, and a custom dimension, etc. The time dimension can be a test time corresponding to the current test, which can be a moment or a time period. The sample dimension can be different sample libraries, and one sample library can correspond to one sample dimension. The experiment dimension can be determined according to the number of experimental groups. When the experimental groups are multiple, each experiment can correspond to a dimension. Specifically, the corresponding experiment dimension can be determined based on the selection operation of the user on multiple test content pushing strategies. The custom dimension can be a dimension customized by the user for analysis, which can be at least one of a region, a country, or a data type, etc.

[0035] In an embodiment, the number of target dimensions can be multiple, and the multi-dimensional operation indicator value can be determined, and the test of the test content pushing strategy can be implemented from multiple dimensions.

[0036] S204, the reference content operation record set is divided to obtain a plurality of reference record sub-sets, and the test content operation record set is divided to obtain a plurality of test record sub-sets.

[0037] The division of the content operation record set can include dividing the content operation records contained in the content operation record set into a group, each group being a record sub-set, and a record sub-set including multiple operation records. "Multiple" means at least two.

[0038] In an embodiment, the implementation process of dividing the reference content operation record set can be: obtaining a division number, wherein the division number is less than the preset set number threshold and makes the number of content operation records in each reference record sub-set obtained by division greater than the preset record number threshold, and dividing the reference content operation record set according to the division number, so as to obtain as few reference record sub-sets as possible in the case that the data amount in the reference record sub-set is large enough. The implementation process of dividing the test content operation record set is the same, and will not be repeated here.

[0039] In an embodiment, the test of the test content pushing strategy can be implemented based on the index value corresponding to each set. The data amount in each set is large enough to make the obtained index value the result of comprehensively considering as many operation record information as possible, guarantee the reliability of the index value, and in addition, the set level index value is obtained, and the set level is greatly reduced in data amount compared with the content operation record level, thereby greatly reducing the data amount of the index value and improving the test efficiency of the content pushing strategy.

[0040] In an embodiment, taking the division method as an example of the bucket division, the reference content operation record set can be divided into buckets, and each bucket obtained is taken as a reference record sub-set. Similarly, the test content operation record set can also be divided into buckets by the bucket method, and each bucket obtained is taken as a reference record sub-set.

[0041] In an embodiment, taking the reference content operation record set as an example, the implementation process of dividing the reference content operation record set can be as shown in Figure 3 302 represents a reference user, Figure 3 one reference content operation record (in other embodiments, one reference user can correspond to multiple reference content operation records), and the corresponding business log can be generated according to each reference content operation record, and the reference content operation record set is generated based on the business log. The reference content operation record set is divided into buckets to divide the reference content operation records in the reference content operation record set into at least one bucket, and the buckets are combined together as a reference record sub-set. The implementation process of dividing the test content operation record set is the same, and will not be repeated here.

[0042] In an embodiment, the division method of the reference content operation record set and the test content operation record set can be consistent or inconsistent. Further, the number of reference record sub-sets and test record sub-sets obtained by division can be the same, so as to facilitate subsequent difference comparison of index values.

[0043] In an embodiment, before the division processing, a process of cleaning the reference content operation records in the reference content operation record set and filtering the abnormal content operation records in the reference content operation record set can also be included. Similarly, before the division processing, a process of cleaning the test content operation records in the test content operation record set and filtering the abnormal content operation records in the test content operation record set can also be included. The process of cleaning the content operation records can include processing invalid or missing content operation records, etc. The abnormal content operation records refer to records with abnormal operations, which can be records with abnormal operation frequency or operation time, etc. For example, a user corresponding to a terminal requests many times of advertisements or clicks many times of the same advertisement within a certain period of time, which can be considered as non-manual operation of these content operations, and thus the corresponding content operation records are determined as abnormal content operation records, and the abnormal content operation records are filtered out.

[0044] S206, sampling the reference record sub-sets multiple times to obtain the sampling reference record sub-set corresponding to each sampling, and sampling the test record sub-sets multiple times to obtain the sampling test record sub-set corresponding to each sampling.

[0045] Among them, the sampling can at least refer to two levels of sampling: the first level is to sample at the reference record sub-set level, that is, to select a certain number of record sub-sets from the record sub-sets and filter out the remaining record sub-sets; the first level is to sample at the content operation record level in the reference record sub-set, that is, to select a certain number of content operation records in each record sub-set and filter out the remaining content operation records.

[0046] In an embodiment, multiple sampling can be realized by the method of resampling. Resampling is a non-parametric method of statistical inference, which extracts repeated samples from the original data sample set and uses empirical methods to generate unique sampling distribution based on actual data. It can be understood as obtaining unbiased estimates based on unbiased samples of all possible outcomes of data.

[0047] In one embodiment, at least one of the resampling methods such as the nearest neighbor method, the bilinear method, the bicubic method, or the Jackknife method can be employed. The Jackknife is a resampling method in statistics used to calculate the variance of a statistic, reducing the bias of the estimate. The variance is a measure of how much the values of a random variable deviate from their mathematical expectation, and is the sum of the squares of the differences between each sample value and the mean of all sample values divided by the number of samples. It is often used to reflect the dispersion of a random variable. The random variable is a real-valued single-valued function of the various outcomes of a random experiment, and is a quantitative expression of a random event. According to whether the values taken within a certain interval are finite or countable, the random variable can be divided into two basic types: discrete random variables and continuous random variables. In one embodiment, the reference content operation records and the test content operation records can be random variables.

[0048] In one embodiment, the multiple sampling of the reference record subsets can include at least one filtering of each reference record subset, filtering at least one reference content operation record each time, and the number of samplings can be less than or equal to the number of reference content operation records in the reference record subsets. The multiple sampling of the reference record subsets can also include at least one filtering of the multiple reference record subsets, filtering at least one reference record subset each time, and the number of samplings can be less than or equal to the number of reference record subsets.

[0049] In one embodiment, the multiple sampling of the test record subsets can include at least one filtering of each test record subset, filtering at least one test content operation record each time, and the number of samplings can be less than or equal to the number of test content operation records in the test record subsets. The multiple sampling of the test record subsets can also include at least one filtering of the multiple test record subsets, filtering at least one test record subset each time, and the number of samplings can be less than or equal to the number of test record subsets.

[0050] S208, the reference operation indicator value corresponding to the sampled reference record subset is obtained as a sample calculation unit, and the test operation indicator value corresponding to the sampled test record subset is obtained as a sample calculation unit.

[0051] The operation indicator value can include an indicator value related to operation, and can be obtained based on a statistical result of the content operation record. In the content pushing scenario, the operation indicator value can include an indicator value of at least one operation of liking, forwarding, or re-editing the pushed content, and can include at least one of an aggregate indicator (Aggregator) such as exposure, click volume, or consumption, or at least one of a ratio indicator (Calculator) such as click rate (CTR) or conversion rate (CVR). In one embodiment, the aggregate indicator corresponding indicator value can be calculated first, and the ratio indicator corresponding indicator value can be calculated based on the aggregate indicator corresponding indicator value. For example, the click volume, exposure, and conversion volume can be calculated based on the content operation record first, and then the click rate can be calculated according to the following formula: click rate = click volume / exposure, and the conversion rate can be calculated according to the following formula: conversion rate = conversion volume / click volume.

[0052] In one embodiment, taking the sample reference record sub-set as an example, the process of obtaining the reference operation indicator value can be: determining the reference content operation record contained in the sample reference record sub-set, extracting the reference content operation record corresponding to the reference operation indicator, and obtaining the reference operation indicator value according to the statistical value of the extracted reference content operation record, for example: extracting the click operation record corresponding to the click volume indicator, and determining the record number of the click operation record as the click volume. In one embodiment, taking the sample test record sub-set as an example, the process of obtaining the test operation indicator value can be: determining the test content operation record contained in the sample test record sub-set, extracting the test content operation record corresponding to the test operation indicator, and obtaining the test operation indicator value according to the statistical value of the extracted test content operation record.

[0053] In one embodiment, a sample reference record sub-set is taken as a sample calculation unit, and then the reference operation indicator value corresponding to the sample reference record sub-set is obtained, for example: the process of obtaining the click volume corresponding to the sample reference record sub-set can be: determining the click operation record in the sample reference record sub-set, and taking the record number corresponding to the click operation record as the click volume of the sample reference record sub-set. Aggregating the calculation unit to the set level can greatly reduce the data volume relative to the user level data. The processing method of the test record sub-set is the same, and will not be repeated here.

[0054] S210, obtaining a strategy test result corresponding to the test content pushing strategy based on the difference between the reference operation indicator value and the test operation indicator value.

[0055] The difference between the reference operation index value and the test operation index value can include at least one of a difference value or a ratio value between the reference operation index value and the test operation index value. In an embodiment, the difference between the reference operation index value and the test operation index value under each sample calculation unit can be determined respectively, and a plurality of differences corresponding to the reference operation index value and the test operation index value can be obtained. Based on the plurality of differences, a strategy test result corresponding to the test content pushing strategy is obtained.

[0056] The strategy test result corresponding to the test content pushing strategy can be at least one of valid or invalid. For valid, the difference between the test content pushing strategy and the reference content pushing strategy is significant, and the newly determined content pushing strategy has a positive or negative effect on the historical content pushing strategy. Specifically, in terms of index values, the newly determined content pushing strategy has a significant difference from the historical content pushing strategy in index values. For example, the click rates corresponding to the two strategies have a large fluctuation, that is, the newly determined content pushing strategy affects the click rate of the user. At this time, the server can consider the test content pushing strategy as valid. For invalid, the difference between the test content pushing strategy and the reference content pushing strategy is not significant, and the newly determined content pushing strategy cannot have a significant effect on the historical content pushing strategy. Specifically, in terms of index values, the newly determined content pushing strategy has a significant difference from the historical content pushing strategy in index values. For example, the click rates corresponding to the two strategies are almost equal or fluctuate within a normal fluctuation range, that is, the newly determined content pushing strategy has little effect on the click rate of the user. At this time, the test server can consider the test content pushing strategy as invalid.

[0057] In an embodiment, content pushing can be performed according to different strategy test results. For example, when the strategy test result is valid and has a positive effect, the test content pushing strategy is promoted to the reference users corresponding to the reference content pushing strategy to realize full-volume pushing, that is, all sample users are pushed content according to the test content pushing strategy. When the strategy test result is invalid, a new test content pushing strategy is determined, and a test is performed on the new test content pushing strategy until an effective test content pushing strategy having a positive effect is found.

[0058] In the traditional method of testing a strategy, such as T-test and Z-test, data needs to be aggregated to the user granularity for further processing when calculating variance. In this case, a large number of data records are involved. If each strategy is tested in real time, it will bring huge performance pressure to the system, and the response time is relatively long, which is difficult for users to accept.

[0059] In the content pushing strategy processing method, the reference content operation record set corresponding to the reference content pushing strategy and the test content operation record set corresponding to the test content pushing strategy are divided respectively, and the record sub-sets obtained by the division are sampled multiple times, and the sub-sets obtained by the sampling realize the reorganization of the content operation record set. The reference operation index value and the test operation index value corresponding to the sub-set obtained by the sampling are obtained at the set granularity. At this time, the reference operation index value and the test operation index value are relative to the data corresponding to the reference content operation record set and the test content operation record set. Therefore, by aggregating the data to the set granularity, the data scale can be greatly reduced. Compared with the traditional method of testing the strategy by the user granularity data, the embodiment of the application can quickly obtain the strategy test result corresponding to the test content pushing strategy based on the difference between the reference operation index value and the test operation index value with small data scale, effectively improve the processing efficiency of the content pushing strategy, greatly reduce the response time of the strategy test, and then realize the real-time output of the strategy test result corresponding to the test content pushing strategy. In addition, the embodiment of the application perfectly combines statistical theory and engineering implementation, supports real-time multi-dimensional variance calculation capability, and can significantly reduce the processing data scale and calculation complexity, save a large amount of machine resources required for storage and calculation, and effectively improve the robustness, flexibility and expandability of the related data flow framework, ensuring that the user can efficiently and scientifically complete data analysis and effect evaluation.

[0060] In one embodiment, the reference record sub-sets are sampled multiple times to obtain a sampling reference record sub-set corresponding to each sampling, including: at each sampling, filtering at least one reference record sub-set from the plurality of reference record sub-sets to obtain a filtered reference record sub-set, and obtaining the sampling reference record sub-set based on the filtered reference record sub-set.

[0061] In one embodiment, the reference record sub-sets are sampled multiple times to obtain a sampling reference record sub-set corresponding to each sampling, including: at each sampling, filtering at least one reference record sub-set from the plurality of reference record sub-sets to obtain a filtered reference record sub-set, and obtaining the sampling reference record sub-set based on the filtered reference record sub-set.

[0062] The implementation process of determining to use the reference record sub-set in the embodiment is illustrated as follows: assuming that the multiple reference record sub-sets constitute a set A = [A1, A2, …, Ai, …, An], where i = 1, 2, 3, 4, …, n. At the first sampling, A1 is filtered out, and the filtered reference record sub-set obtained is A1' = [A2, A3, …, Ai, …, An]. At the second sampling, A2 is filtered out, and the filtered reference record sub-set obtained is A2' = [A1, A3, …, Ai, …, An]. At the nth sampling, An is filtered out, and the filtered reference record sub-set obtained is An' = [A1, A2, …, Ai, …, An-1]. The sampling reference record sub-set AA = [A1', A2', …, An'] is obtained based on the filtered reference record sub-sets A1', A2', …, An'.

[0063] In the above embodiment, the reference record sub-sets are sampled multiple times, and at each sampling, at least one reference record sub-set is filtered out, that is, discarded. The number of filtered reference record sub-sets is reduced. The index value is obtained based on the set granularity, so that the data amount of the reference operation index value corresponding to the sampling reference record sub-set is reduced, the data amount to be processed in the strategy test process is reduced, and the strategy test efficiency is improved.

[0064] In one embodiment, the test record sub-sets are sampled multiple times to obtain a sampling test record sub-set corresponding to each sampling, including: at each sampling, filtering out at least one test record sub-set from the multiple test record sub-sets to obtain a filtered test record sub-set, and obtaining a sampling test record sub-set based on the filtered test record sub-set.

[0065] At least one test record sub-set can be filtered out from the multiple test record sub-sets, and the number of filtered test record sub-sets is reduced by at least one compared with the number of unfiltered test record sub-sets. The test record sub-sets can be filtered out in order or not in order. For the case of filtering out the test record sub-sets in order, the first test record sub-set in the multiple test record sub-sets can be filtered out at the first sampling, the second test record sub-set in the multiple test record sub-sets can be filtered out at the second sampling, and so on until the sampling number meets the requirement. The sampling number can be any value less than or equal to the number of test record sub-sets.

[0066] The implementation process of determining to use the test record sub-sets in the embodiment is described as follows: assuming that the test record sub-sets constitute a set B = [B1, B2, …, Bi, …, Bn]. In the first sampling, B1 is filtered out, and the filtered test record sub-set B1' = [B2, B3, …, Bi, …, Bn] is obtained. In the second sampling, B2 is filtered out, and the filtered test record sub-set B2' = [B1, B3, …, Bi, …, Bn] is obtained. In the i-th sampling, Bi is filtered out, and the filtered test record sub-set Bi' = [B1, B2, …, Bi-1, Bi+1, …, Bn] is obtained. In the n-th sampling, Bn is filtered out, and the filtered test record sub-set Bn' = [B1, B2, …, Bi, …, Bn-1] is obtained. The sampling test record sub-set BB = [B1', B2', …, Bn'] is obtained based on the filtered test record sub-sets B1', B2', …, Bn'.

[0067] In the above embodiment, the test record sub-sets are sampled for multiple times, and at least one test record sub-set is filtered out in each sampling, that is, at least one test record sub-set is discarded, the number of filtered test record sub-sets is reduced, the index value is obtained based on the set granularity, the data amount of the test operation index value corresponding to the sampling test record sub-set is reduced, the data amount to be processed in the strategy test process is reduced, and the strategy test efficiency is improved.

[0068] In one embodiment, when the filtered reference record sub-sets are obtained, the reference content operation records in the filtered reference record sub-sets can be further filtered out.

[0069] In one embodiment, obtaining the sampling reference record sub-sets based on the filtered reference record sub-sets includes: filtering out at least one reference content operation record from at least one reference record sub-set of the filtered reference record sub-sets, and combining the reference record sub-sets after the reference content operation records are filtered out to obtain the sampling reference record sub-sets.

[0070] The implementation process of combining the reference record sub-sets after the reference content operation records are filtered out to obtain the sampling reference record sub-sets can be as follows: for any one reference record sub-set C1 in the filtered reference record sub-sets, the reference content operation records remaining after the filtering are combined together as the sampling reference record sub-set corresponding to the reference record sub-set C1.

[0071] The implementation process of determining to use the reference record sub-set in the embodiment is illustrated as follows: for the filtered reference record sub-set B1' = [B1, B2, …, Bi-1, Bi+1, …, Bn], wherein B1 = [b11, b12, …, b1j, …, b1m], …, Bn = [bn1, bn2, …, bnj, …, bnm]. Taking B1 as an example, as shown in the following table, the implementation process of sampling B1 can be as follows: at the first sampling, b1 is filtered out, and the new reference record sub-set obtained is b1' = [b12, b13, …, b1i, …, b1n]; at the second sampling, b2 is filtered out, and the new reference record sub-set obtained is b2' = [b21, b23, …, b2i, …, b2n]; at the i-th sampling, bi is filtered out, and the new reference record sub-set obtained is bi' = [bi1, bi2, …, bii-1, bii+1, …, bin]; and at the n-th sampling, bn is filtered out, and the new reference record sub-set obtained is bn' = [bn1, bn2, …, bni, …, bnn-1]. The new record sub-set bb1 corresponding to B1 is obtained by combining the new reference record sub-sets b1', b2', …, bn'. Figure 4

[0072] As shown in the following table, the data amount of the reference content operation record in a new reference record sub-set is greatly increased compared with that in an original reference record sub-set, that is, the data amount corresponding to a set is greatly improved, which can effectively improve the reliability of the reference operation index value obtained from the set, and further improve the test accuracy of the content pushing strategy. Figure 4

[0073] In the above embodiment, the filtering processing is performed in the sub-set and content operation record dimensions respectively, which can compress the data amount from multiple aspects and effectively improve the efficiency of the strategy test.

[0074] In one embodiment, when the filtered test record sub-set is obtained, the test content operation record in the filtered test record sub-set can be further filtered out.

[0075] In one embodiment, obtaining the sampling test record sub-set based on the filtered test record sub-set includes: filtering out at least one test content operation record from at least one test record sub-set of the filtered test record sub-set, and combining the test record sub-sets after filtering out the test content operation record to obtain the sampling test record sub-set.

[0076] ​​The implementation process of combining each test record sub-set after filtering the test content operation records to obtain the sampling test record sub-set can be as follows: for any one test record sub-set C2 in the filtered test record sub-set, each test content operation record remaining after the filtering is combined together as the sampling test record sub-set corresponding to the test record sub-set C2.

[0077] The implementation process of determining the test record sub-set in the embodiment is described as follows: for the filtered test record sub-set Ai' = [A1, A2, …, Ai-1, Ai+1, …, An], where A1 = [a11, a12, …, a1j, …, a1m], …, An = [an1, an2, …, anj, …, anm]. Taking A1 as an example, the implementation process of sampling A1 can be as follows: in the first sampling, a1 is filtered to obtain the filtered test record sub-set a1' = [a12, a13, …, a1i, …, a1n]; in the second sampling, a2 is filtered to obtain the new test record sub-set a2' = [a21, a23, …, a2i, …, a2n]; …; in the i-th sampling, ai is filtered to obtain the new test record sub-set ai' = [ai1, ai2, …, aii-1, aii+1, …, ain]; …; in the n-th sampling, an is filtered to obtain the new test record sub-set an' = [an1, an2, …, ani, …, ann-1]. The new record sub-set aa1 corresponding to A1 is obtained by combining the new test record sub-sets a1', a2', …, an'. The sampling test record sub-set corresponding to the filtered test record sub-set Ai' is obtained by combining the new record sub-sets corresponding to each test record sub-set in the filtered test record sub-set Ai'.

[0078] In the above embodiment, the filtering is performed in the sub-set and content operation record dimensions, which can compress the data amount from multiple aspects and effectively improve the efficiency of the strategy test.

[0079] In one embodiment, the reference content operation record set is divided to obtain a plurality of reference record sub-sets, including: obtaining attribute information of a reference user corresponding to a reference content operation record; obtaining a division feature value corresponding to the reference content operation record based on the attribute information of the reference user; and dividing the reference content operation record set based on the division feature value corresponding to the reference content operation record to obtain a plurality of reference record sub-sets.

[0080] The attribute information of the reference user can include at least one of a name, a gender, an age, a preference, or a geographic location of the reference user. The attribute information of the reference user can be represented by at least one of a character, a numerical value, or a symbol. In an embodiment, the attribute information of the reference user can further include a content operation record of the reference user on the reference push content corresponding to the reference content push strategy and a number of the content operation record, so as to obtain a division feature value and then divide the reference content operation record set. The number of the content operation record of the reference push content can be used to determine the activity level of the corresponding reference user, and the reference record sub-set obtained by dividing the reference content operation record set according to the activity level is sufficient to consider the activity level, so that the activity level corresponding to each reference record sub-set remains in a balanced state.

[0081] The division feature value refers to a feature value related to division, which can include a data string that can be used for division processing. By operating the data string, the reference content operation record set can be divided based on the operation result. The division feature value can also be a feature value used for grouping identification. Different feature values can correspond to different groups, so as to divide the reference content operation record set into different groups. For example, when the division feature value is 0.1, the corresponding reference content operation record set is divided into the first group; when the division feature value is 0.2, the corresponding reference content operation record set is divided into the second group; when the division feature value is 0.9, the corresponding reference content operation record set is divided into the ninth group, and so on.

[0082] In an embodiment, the division feature value corresponding to the reference content operation record is obtained based on the attribute information of the reference user, including: converting the attribute information of the reference user into a reference string, obtaining a hash value corresponding to the reference string, and obtaining the division feature value corresponding to the reference content operation record according to the calculated hash value.

[0083] In the above embodiment, the reference content operation record set is divided based on the attribute information of the reference user, and a plurality of reference record sub-sets are obtained after the attribute information of the reference user is fully considered. The attribute information of the users in the obtained reference record sub-set is as balanced as possible, thereby reducing the difference between different reference record sub-sets. The server can process each reference record sub-set according to similar operation logic, improve the efficiency of obtaining the reference operation index value, and then improve the strategy test efficiency.

[0084] In an embodiment, the buckets are divided in a manner such that the attribute information of the users in each bucket after processing is as uniform as possible, for example, so that the male-female ratio in each bucket is not unbalanced. In addition, the content operation records related to the same user are placed in one bucket, and here it is possible that A user corresponds to 10 content operation records and is divided into bucket 1, and B user corresponds to 100 content operation records and is divided into bucket 2. To avoid the situation that the number of content operation records in bucket 1 and bucket 2 is not uniform due to this, the user activity is considered when dividing the buckets, and the number of content operation records is a manifestation of the user activity. After dividing the buckets, the activity of each bucket is as uniform as possible, which also makes the number of content operation records in each bucket uniform.

[0085] In an embodiment, the division feature value corresponding to the reference content operation record is obtained based on the attribute information of the reference user, including: performing feature mapping based on the attribute information of the reference user to obtain a feature mapping value corresponding to the reference content operation record; obtaining a division number corresponding to the reference content operation record set; performing modulo calculation based on the feature mapping value and the division number to obtain the division feature value corresponding to the reference content operation record.

[0086] The feature mapping refers to mapping the attribute information of the reference user to a numerical value according to a specific rule, and taking the obtained numerical value as the feature mapping value corresponding to the reference content operation record. The feature mapping can be a hash operation, a modulo operation based on the hash operation, etc. The hash operation can be at least one of division hash, square hash, Fibonacci hash, or Murmur Hash (a non-encryption hash function), etc. The Murmur Hash is a non-encryption hash function.

[0087] The division number can be determined according to the number of reference users corresponding to the reference content pushing strategy. In the case of ensuring the number of reference content operation records or the number of reference users in the reference record sub-set obtained by division, the division number can be as small as possible. In an embodiment, the number of reference content operation records in the reference record sub-set can be in the order of millions to improve the reliability of the reference operation index value corresponding to each reference record sub-set.

[0088] In an embodiment, the division number can be calculated by taking the modulus of the feature mapping value corresponding to any reference content operation record C3, and the result of the modulo calculation is determined as the division feature value corresponding to the reference content operation record.

[0089] The above embodiment determines the division feature value through feature mapping and modulo calculation, and can thus simply and uniformly divide the reference content operation record set into multiple reference record sub-sets, thereby improving the processing efficiency of the content pushing strategy.

[0090] In one embodiment, dividing the test content operation record set to obtain multiple test record sub-sets includes: obtaining attribute information of a test user corresponding to a test content operation record; obtaining a division feature value corresponding to the test content operation record based on the attribute information of the test user; and dividing the test content operation record set based on the division feature value corresponding to the test content operation record to obtain multiple test record sub-sets.

[0091] The attribute information of the test user can include at least one of the name, gender, age, preference, or geographic location of the test user. The attribute information of the test user can be represented by at least one of text, numerical value, or symbol. In one embodiment, the attribute information of the test user can also include the content operation record of the test user on the test push content corresponding to the test content pushing strategy and the number of content operation records, etc., to obtain the division feature value and further divide the test content operation record set. The number of content operation records of the test push content can be used to determine the activity level of the corresponding test user, and the test record sub-sets obtained by dividing the test content operation record set take into account the activity level, so that the activity level of each test record sub-set remains balanced.

[0092] The division feature value refers to a feature value related to division, which can include a data string that can be used for division processing. By operating the data string, the test content operation record set can be divided based on the operation result. It can also be a feature value used for grouping identification. Different feature values can correspond to different groups to divide the test content operation record set into different groups, for example: when the division feature value is 0.1, the corresponding test content operation record set is divided into the first group, when the division feature value is 0.2, the corresponding test content operation record set is divided into the second group, when the division feature value is 0.9, the corresponding test content operation record set is divided into the ninth group, and so on.

[0093] In one embodiment, obtaining the division feature value corresponding to the test content operation record based on the attribute information of the test user includes: converting the attribute information of the test user into a test string, obtaining a hash value corresponding to the test string, and obtaining the division feature value corresponding to the test content operation record according to the calculated hash value.

[0094] In the above embodiments, the test content operation record set is divided based on the attribute information of the test user, and a plurality of test record sub-sets are obtained after fully considering the attribute information of the test user. The attribute information of the users in the obtained test record sub-set is as balanced as possible, thereby reducing the difference between different test record sub-sets. The server can process each test record sub-set according to similar operation logic, improve the efficiency of obtaining the test operation index value, and further improve the strategy test efficiency.

[0095] In one embodiment, the division feature value corresponding to the test content operation record is obtained based on the attribute information of the test user, including: performing feature mapping based on the attribute information of the test user to obtain a feature mapping value corresponding to the test content operation record; obtaining a division number corresponding to the test content operation record set; and performing modulo calculation based on the feature mapping value and the division number to obtain the division feature value corresponding to the test content operation record.

[0096] The feature mapping refers to mapping the attribute information of the test user according to a specific rule to obtain a numerical value, and the obtained numerical value is used as the feature mapping value corresponding to the test content operation record. The feature mapping can be a hash operation, a modulo operation based on the hash operation, etc.

[0097] The division number can be determined according to the number of test users corresponding to the test content pushing strategy. In the case of ensuring the number of test content operation records or the number of test users in the test record sub-set obtained by division, the division number can be as small as possible. In one embodiment, the number of test content operation records in the test record sub-set can be in the order of millions to improve the reliability of the test operation index value corresponding to each test record sub-set.

[0098] In one embodiment, the division number can be calculated by performing modulo operation based on the feature mapping value corresponding to any test content operation record C4, and the result of the modulo calculation is determined as the division feature value corresponding to the test content operation record.

[0099] In the above embodiments, the division feature value is determined by feature mapping and modulo calculation, which can be used to simply and uniformly divide the test content operation record set into a plurality of test record sub-sets, thereby improving the processing efficiency of the content pushing strategy.

[0100] In an embodiment, the strategy test result corresponding to the test content pushing strategy is obtained based on the difference between the reference operation indicator value and the test operation indicator value, including: comparing the test operation indicator value matched with the sampling sequence with the reference operation indicator value to obtain an indicator improvement value of the test operation indicator value relative to the reference operation indicator value, and each indicator improvement value constitutes an indicator improvement value set; performing statistics on the indicator improvement value set to obtain an indicator statistical value; obtaining a statistical distribution range corresponding to the indicator improvement value based on the difference between each indicator improvement value and the indicator statistical value; comparing the statistical distribution range with a range threshold, and obtaining the strategy test result corresponding to the test content pushing strategy based on the comparison result.

[0101] In an embodiment, the implementation process of comparing the test operation indicator value matched with the sampling sequence with the reference operation indicator value can be: comparing the test operation indicator value of each sampling with the reference operation indicator value under the same sampling sequence according to the sampling sequence, for example: comparing the test operation indicator value corresponding to the first sampling with the reference operation indicator value corresponding to the first sampling, and comparing the test operation indicator value corresponding to the second sampling with the reference operation indicator value corresponding to the second sampling; or comparing the test operation indicator value under different sampling sequences with the reference operation indicator value, for example: comparing the test operation indicator value corresponding to the first sampling with the reference operation indicator value corresponding to the second sampling.

[0102] In an embodiment, the indicator improvement value of the test operation indicator value relative to the reference operation indicator value can be calculated by dividing the difference between the test operation indicator value and the reference operation indicator value by the reference operation indicator value.

[0103] In an embodiment, the statistics on the indicator improvement value set can be mean calculation on the indicator improvement value, and the calculated mean value is taken as the indicator statistical value. The mean value represents the measure value of the trend in a group of data, which is the result value obtained by dividing the sum of all data in a group of data by the number of data, and is usually used to reflect the general level and concentration degree of the data set.

[0104] The statistical distribution range can be a distribution range of the index improvement value, can be a main distribution interval of the index improvement value, that is, a distribution range corresponding to most index improvement values, for example, a distribution range of the index improvement value under a confidence probability threshold. The confidence probability threshold refers to a probability threshold for evaluating the data distribution range, and its size can be determined according to actual conditions. In an embodiment, the confidence probability threshold can also be determined according to a significance level, for example, when the significance level is 0.05, the confidence probability threshold is determined as 95%. The size of the significance level can be determined by the server according to the requirement for significance accuracy in the current test process. When the requirement for significance accuracy is high, the significance level can be determined as a larger value, for example, 0.1. When the requirement for significance accuracy is low, the significance level can be determined as a smaller value, for example, 0.02.

[0105] The range threshold can be a threshold for evaluating the interval distribution of the statistical distribution range. The range threshold can be 0, 0.2, etc.

[0106] In an embodiment, comparing the statistical distribution range with the range threshold can be comparing the boundary value of the statistical distribution range with the range threshold, and then determining whether the range threshold is in the statistical distribution range.

[0107] In the above embodiment, the strategy test result corresponding to the test content pushing strategy is determined based on the statistical distribution range, which can quickly and simply determine the strategy test result based on a statistical method, and improves the processing efficiency of the content pushing strategy.

[0108] In an embodiment, the strategy test result of the target content pushing strategy in the target dimension is obtained based on the difference between the reference operation index value and the target operation index value, including: calculating the improvement rate of the target operation index value relative to the reference operation index value based on the difference between the reference operation index value and the target operation index value in each sample calculation unit; performing statistics on the index improvement value set to obtain an index statistical value; comparing the statistical distribution range with the range threshold, and obtaining the strategy test result corresponding to the test content pushing strategy based on the comparison result.

[0109] In an embodiment, the index statistical value is obtained by performing statistics on the index improvement value set, including: obtaining the improvement rate in each sample calculation unit, and determining the mean of the improvement rate in each sample calculation unit; determining the variance of the improvement rate according to the mean, and determining the variance as the index statistical value.

[0110] In an embodiment, the range threshold is zero, and the strategy test result corresponding to the test content pushing strategy is obtained according to comparison of the statistical distribution range with the range threshold based on a comparison result, including: obtaining a significance level; constructing a confidence interval corresponding to the significance level according to the index statistical value; when the confidence interval does not contain zero, determining that the target content pushing strategy is significantly different from the reference content pushing strategy, and generating a strategy test result that the target content pushing strategy is effective; when the confidence interval contains zero, determining that the target content pushing strategy is not significantly different from the reference content pushing strategy, and generating a strategy test result that the target content pushing strategy is ineffective.

[0111] In an embodiment, the statistical distribution range corresponding to the index improvement value based on the difference between the index improvement value and the index statistical value includes: obtaining an improvement value dispersion corresponding to the index improvement value based on the difference between the index improvement value and the index statistical value; obtaining a test coefficient corresponding to a confidence probability threshold; performing product calculation on the test coefficient and the improvement value dispersion to obtain a test deviation value; and obtaining the statistical distribution range corresponding to the index improvement value according to the index statistical value and the test deviation value.

[0112] In an embodiment, the range threshold can be an evaluation value corresponding to the test content pushing strategy and the reference content pushing strategy being equal. The strategy test result can be determined based on the size relationship between the statistical distribution range and the range threshold, when the range threshold is within the statistical distribution range, it is determined that the test content pushing strategy and the reference content pushing strategy have no obvious difference, and the strategy test result is determined as the test content pushing strategy being ineffective, when the range threshold is not within the statistical distribution range, it is determined that the test content pushing strategy and the reference content pushing strategy have obvious difference, and the strategy test result is determined as the test content pushing strategy being effective.

[0113] The improvement value dispersion can include information related to the dispersion degree of the index improvement value, and can be represented by at least one of variance or standard deviation. The test coefficient can be a test coefficient obtained by operating the confidence probability threshold. For example, when the confidence probability threshold is 95%, the test coefficient can be 1.96.

[0114] In an embodiment, the implementation manner of obtaining the statistical distribution range corresponding to the index improvement value according to the index statistical value and the test deviation value can be: taking the difference between the index statistical value and the test deviation value as the minimum value of the statistical distribution range, taking the sum of the index statistical value and the test deviation value as the maximum value of the statistical distribution range, and determining the interval between the maximum value and the minimum value as the statistical distribution range.

[0115] In the above embodiments, the strategy test result corresponding to the test content pushing strategy is determined based on the statistical distribution range, which can quickly and simply determine the strategy test result based on statistical methods, and improve the processing efficiency of the content pushing strategy.

[0116] In an embodiment, the comparison between the statistical distribution range and the range threshold value is used to obtain the policy test result corresponding to the test content pushing strategy, including: comparing the minimum value of the statistical distribution range with the range threshold value to obtain a comparison result; when the comparison result is that the minimum value is greater than the range threshold value, determining that the test result of the test content pushing strategy is valid, and taking the test content pushing strategy as a target content pushing strategy to perform content pushing using the target content pushing strategy.

[0117] In an embodiment, the index improvement value corresponding to the confidence probability threshold value falls within the statistical distribution range. Assuming that the range threshold value is 0. When the minimum value of the statistical distribution range is greater than 0, it indicates that the index improvement value corresponding to the confidence probability threshold value is all greater than 0, at this time, it indicates that the test operation index value is positively improved relative to the reference operation index value, that is, the test content pushing strategy has a positive effect, at this time, the server determines that the policy test result is that the test content pushing strategy is valid. Therefore, the test content pushing strategy can be determined as the target content pushing strategy, and content pushing is performed using the target content pushing strategy.

[0118] In an embodiment, taking an advertisement pushing scenario as an example, the corresponding terminal can be controlled to display an advertisement corresponding to the target content pushing strategy on an interface. As shown in FIG. 5, the pushed advertisement is displayed in the interface 502 of the terminal in the form of a page. Figure 5

[0119] In the above embodiment, the policy test result is determined according to the comparison result of the statistical distribution range and the range threshold value, the target content pushing strategy that has a significant difference from the reference content pushing strategy is determined in a simple and intuitive manner, it is ensured that the subsequent content pushing has a significant effect relative to the historical content pushing, rather than fluctuating within the sampling error range, and it is prevented that the subsequent content pushing effect is not obvious.

[0120] In an embodiment, the content pushing strategy processing method further includes: when the minimum value of the statistical distribution range is less than or equal to the range threshold value, determining that the test content pushing strategy has a reverse effect, discarding the test content pushing strategy, obtaining a new test content pushing strategy, testing the new test content pushing strategy based on the reference content pushing strategy, and determining the target content pushing strategy according to the policy test result corresponding to the new test content pushing strategy to perform content pushing using the target content pushing strategy. In the above embodiment, when the test content pushing strategy has a reverse effect, a new test content pushing strategy is determined, and the target content pushing strategy having a significant effect relative to the reference content pushing strategy can be determined, thereby improving the subsequent content pushing effect.

[0121] ​In an embodiment, the statistical distribution range can be represented by a confidence interval. When the confidence interval does not contain zero, it is determined that the target content pushing strategy is significantly different from the reference content pushing strategy; when the confidence interval contains zero, it is determined that the target content pushing strategy is not significantly different from the reference content pushing strategy.

[0122] In an embodiment, the reference content operation record corresponding to the reference content operation record set is obtained, and the test content operation record corresponding to the test content operation record set is obtained, including: determining a reference user and a test user from the sample users; obtaining historical pushing content for the target pushing resource as reference pushing content matched by the reference content pushing strategy, and obtaining current pushing content for the target pushing resource as test pushing content matched by the test content pushing strategy; pushing the reference pushing content to the terminal of the reference user to obtain the reference content operation record in response to the operation of the reference user on the reference pushing content; and pushing the test pushing content to the terminal of the test user to obtain the test content operation record in response to the operation of the test user on the test pushing content.

[0123] The target pushing resource refers to a resource corresponding to the pushing content. Taking an advertisement as an example, the target pushing resource can include a product or a service corresponding to the advertisement, etc.

[0124] In an embodiment, taking an advertisement scenario as an example, for a certain product P, the process of determining the reference pushing content can be: obtaining an old advertisement pushing strategy for the product P as the reference pushing content. The process of determining the test pushing content can be: obtaining a newly determined advertisement pushing strategy for the product P as the test pushing content.

[0125] In an embodiment, the reference content operation record set corresponding to the reference content pushing strategy is obtained, including: obtaining historical pushing content for the target pushing resource as reference pushing content matched by the reference content pushing strategy, pushing the reference pushing content to the terminal of the reference user to obtain the reference content operation record in response to the operation of the reference user on the reference pushing content, and obtaining the reference content operation record set according to the reference content operation record.

[0126] In an embodiment, the test content operation record set corresponding to the test content pushing strategy is obtained, including: obtaining current pushing content for the target pushing resource as test pushing content matched by the test content pushing strategy; pushing the test pushing content to the terminal of the test user to obtain the test content operation record in response to the operation of the test user on the test pushing content; and obtaining the test content operation record set according to the test content operation record.

[0127] In the above embodiments, the reference push content corresponding to the old content push strategy is determined, and the test push content corresponding to the new content push strategy is determined, so that different content can be pushed to different users, the content operation record comparison of different content push strategies is realized, and the test of the new content push strategy is realized.

[0128] In one embodiment, the method further includes: when the policy test result is valid, taking the test content push strategy as a target content push strategy; the push content corresponding to the target content push strategy is a target push content; obtaining a target content feature corresponding to the target push content; obtaining a corresponding relationship between the content feature and the feature of the user in the target content push strategy; obtaining the feature of the target user based on the corresponding relationship and the target content feature, obtaining a target user corresponding to the feature of the target user, and pushing the target push content to a terminal corresponding to the target user.

[0129] In the above embodiments, the content feature can include a feature used for distinguishing the target push content, and can be implemented by numbering or coding, etc. The coding can be obtained by coding conversion on data corresponding to the target push content, for example, at least one of the video frame rate, video length or video resolution of an advertisement video is converted into a vector, and the converted vector is taken as the target content feature corresponding to the advertisement video.

[0130] The feature of the user can include a feature used for distinguishing the user from each user, and can be determined by a user account or attribute information of the user, etc. Further, the feature of the user can also include a user portrait, so that the target user whose portrait matches the target content feature can be pushed the target push content, the accurate push of the push content is realized, the personalized push is realized, and the push efficiency of the content is improved.

[0131] In the above embodiments, when the test content push strategy is determined to be a valid content push strategy, the target user is determined based on the corresponding relationship between the content feature and the feature of the user, and the test content push strategy is pushed to the terminal corresponding to the target user, so that the content can be pushed based on the test result of the push strategy, and the content pushed to the target user can have good push effect.

[0132] In one embodiment, obtaining the reference content operation record set corresponding to the reference content push strategy and the test content operation record set corresponding to the test content push strategy includes: obtaining dimension selection information sent by the target terminal, and obtaining a target dimension corresponding to the dimension selection information; obtaining a content operation record matched with the target dimension in the content operation record corresponding to the reference content push strategy to obtain the reference content operation record set; obtaining a test content operation record matched with the target dimension in the content operation record corresponding to the test content push strategy to obtain the test content operation record set.

[0133] The target terminal can be a terminal capable of network communication with the server, and can also be referred to as a front end.

[0134] The number of target dimensions can be at least one. When the target dimensions can be multiple, the server can combine the target dimensions to obtain a combined dimension, and then obtain the reference content operation record and the test content operation record corresponding to the combined dimension. For example, the target dimensions are the automobile industry and the spare parts industry, the server combines the target dimensions to obtain a combined dimension of automobile spare parts, and then obtains the reference content operation record and the test content operation record corresponding to the automobile spare parts.

[0135] In the above embodiments, the server obtains the corresponding reference content operation record according to the target dimension selected by the target terminal to obtain a reference content operation record set, and obtains the test content operation record corresponding to the target dimension to obtain a test content operation record set, and realizes the test of the test content push strategy based on the processing of the reference content operation record set and the test content operation record set, and then obtains the strategy test result of the test content push strategy under the target dimension. When the target dimension is multiple, the strategy test result of the test content push strategy corresponding to multiple dimensions can be realized in real time.

[0136] In one embodiment, obtaining the reference content operation record set corresponding to the reference content push strategy and the test content operation record set corresponding to the test content push strategy includes: obtaining the reference content operation record set corresponding to the reference content push strategy and the test content operation record set corresponding to the test content push strategy when the test trigger operation sent by the target terminal is obtained.

[0137] The target terminal can obtain the test trigger operation through a test entry on the interface. In one embodiment, the test entry can be realized through a "query" control on the interface of the target terminal to query the strategy test result of the test content push strategy.

[0138] The server can determine that the query request sent by the target terminal is received when the test trigger operation sent by the target terminal is obtained, obtain the strategy test result of the test content push strategy, and return the query result to the target terminal according to the strategy test result of the test content push strategy.

[0139] In one embodiment, the target terminal can obtain at least one of the dimension selection information, the time selection information, the strategy configuration information, or the test trigger operation based on the interaction with the user. The user can select a strategy ID (Identity Document, unique code) to configure the reference content push strategy and the test content push strategy.

[0140] The interface displayed by the target terminal can be as shown in Figure 6 The target terminal can obtain the industry, time, experiment and index selected by the user through the interface as shown in Figure 6 The target terminal converts the query conditions checked by the user into an SQL (Structured Query Language) statement, and then sends a query request to the server. Assuming that the user selects: time: 2020.01.01, strategy 1, automobile industry, and click rate index, the rendered query statement is: select strategy 1, sum (click volume) from data table where industry = automobile and time = 2020.01.01. The target terminal sends the query request to the server to trigger the server to obtain the strategy test result corresponding to the test content push strategy and return the strategy test result corresponding to the test content push strategy to the target terminal.

[0141] In one embodiment, after obtaining the strategy test result corresponding to the test content push strategy based on the difference between the reference operation index value and the test operation index value, the method further includes: outputting the strategy test result to the target terminal, so that the target terminal displays the strategy test result.

[0142] In one embodiment, in addition to displaying the strategy test result, the target terminal can also display at least one of the reference operation index value, the test operation index value, or the index improvement value. As shown in Figure 7 The target terminal displays the click rate of the experimental group and the control group, the improvement rate of the click rate of the experimental group relative to the control group, and the test result of the experimental group in the interface. In one embodiment, when the experimental group is multiple, the click rate, the improvement rate of the click rate, and the test result of each experimental group can be displayed synchronously in the interface, so that the user can intuitively view the differences of each experimental group in the test.

[0143] In one embodiment, Figure 8 The system structure diagram for implementing the content push strategy processing method in one embodiment is shown in the figure. The modules in the system for implementing the content push strategy processing method are divided into a storage layer, a calculation layer, and an application layer according to their functional characteristics. The calculation layer performs bucketing and resampling processing on the data aggregated to the user level, and then performs calculation of related parameters and verification of index change significance at the bucket level. The other two layers provide input data sources and result display pages for experiment evaluation related calculations. Referring to Figure 8 , the following will be described in detail taking one content push strategy corresponding to one experiment as an example:

[0144] Application layer:

[0145] The application layer can be implemented through a Web (World Wide Web) front end, which provides a service interface, i.e., an effect query API (Application Programming Interface), also referred to as a service interface, to query the test effect of the experimental group. The Web front end and the service interface are described in detail below.

[0146] The Web front end: an entry for users to evaluate the test effect of the strategy, responsible for completing query interaction with the user, constructing interface parameters, and displaying experimental evaluation results. When the Web front end receives interface parameter information input by the user, it can determine the time dimension, experimental dimension, and custom dimension corresponding to the current test. Based on the time dimension, a first content operation record is determined. For example, a content operation record in a certain time period can be obtained as the first content operation record. Further, based on the experimental dimension, a second content operation record is determined. Assuming that the determined experimental dimension is experiment 1, the second content operation record corresponding to experiment 1 can be obtained based on the first content operation record. Further, based on the experimental dimension, a third content operation record is determined. Assuming that the determined custom dimension is region A, the third content operation record corresponding to region A can be obtained based on the second content operation record. According to the third content operation record, a corresponding content operation record set can be constructed.

[0147] The service interface: in response to experimental evaluation request information, converts various query parameters into corresponding query statements and sends them to the effect evaluation framework of the computing layer. After processing the experimental evaluation results, the data is returned to the front end. At this time, the target dimension selected by the user can be obtained.

[0148] The computing layer:

[0149] ETL data flow: extracts records related to test experimental effects from raw log data, pre-processes user samples, for example, integrates multiple operation records corresponding to a user into one content operation record, randomly divides the operation content records obtained through pre-processing into buckets, and based on the bucketing results, calculates the pre-aggregation of operation content records corresponding to each target dimension using each bucket as a sample unit. ETL stands for Extract Transform Load (data warehouse technology).

[0150] Effect evaluation framework: responsible for the core processing flow of strategy testing. After calculating various business indicator values and lift rates, resampling is performed based on the bucketing results to generate a new sample set, complete index calculation, and perform significance testing based on the index calculation results to determine whether there is a significant difference between the experimental group and the control group, and thus obtain the experimental evaluation results of the experimental group.

[0151] Storage layer: mainly the log module of the business system, which stores the log information of various user behaviors, including effect data attributed to the experiment, which can provide raw data for various calculations for experiment evaluation. In the advertising scenario, the effect data can be actual data generated by user operations such as clicking on an advertisement. As shown in Figure 8 The storage layer can be implemented through a log system. At least one of the request log, exposure log, click log, and conversion log is stored in the log system.

[0152] The application also provides an application scenario for the content push strategy processing method. Specifically, as shown in Figure 9 The content push strategy processing method is applied as follows in the application scenario:

[0153] I. User bucketing, resampling, and index calculation

[0154] The following illustrates the processing procedure of user bucketing, resampling, and index calculation using a control group as an example:

[0155] 1. User bucketing:

[0156] After extracting the user sample information corresponding to the content operation record from the original log record, a randomization algorithm is called to randomize according to the attribute information of the user, obtaining the bucket ID corresponding to each user sample. Each bucket contains a number of user sample data, and the number of user samples is uniform among different buckets, finally forming a user bucket set.

[0157] 2. Resampling: Resample the original sample set and generate a new sample set. Suppose there is a user data set X = [x1, x2, x3, …, xi, …, xn], and the index Y = f(x1, x2, x3, …, xi, …, xn) is calculated based on these data. To obtain the variance of the index, each sample can be discarded one by one to form a new sample with the remaining original sample set, and finally form n new samples. That is, for the ith processing, discard the ith sample from x1, x2, x3, …, xi, …, xn, and generate a new sample Xi' = [x1, x2, x3, …, xi-1, xi+1, …, xn] with the remaining n-1 original samples. After n such processes, a new sample set X' = [X1', X2', X3', …, Xi', …, Xn'] is obtained. To further reduce the sample data size, use the bucketing result to discard the data of one bucket each time, and use the remaining bucket data to form a new sample Bi' = [b1, b2, b3, …, bi-1, bi+1, …, bm], and generate a new sample set B' = [B1', B2', B3', …, Bi', …, Bn'].

[0158] After the randomization and bucketing of the users is completed, the bucketing set is resampled to generate new sample sets, and the data corresponding to each new sample set is obtained to calculate the indicators. This not only ensures the independence and randomness of the new samples, but also reduces the large user sample size to a limited number of buckets, greatly reducing the engineering cost.

[0159] 3. Indicator calculation: According to the statistical scope and calculation formula of each business indicator, the data after resampling of the original user sample is used to obtain the indicator value of each experiment. Because the magnitude of the data set after bucketing is greatly reduced, the sample time experiment analysis dimension level data can be stored to complete the experiment indicator calculation in multiple dimensions. As shown in FIG. 3, the actual calculation is performed in two steps in parallel. The first step is to calculate the indicator value and lift rate of the indicator result in the sample time experiment analysis dimension, which is used as the intuitive effect of the experiment for front-end display. The other step is to calculate the indicator value and change rate of the indicator result in the sample time experiment analysis dimension based on the resampling result, as shown in FIG. 4. The lift rate is calculated for each of the n buckets, which is necessary data for the subsequent significance test. The server can return the results obtained in the two steps to the front end. Since the significance test can occur in real time, the front end can display the indicator value, change rate, and test result synchronously. Figure 9 Figure 9

[0160] In one embodiment, as shown in FIG. 2, the total quantity type indicators corresponding to each bucket can be determined, the ratio type indicators can be obtained based on the total quantity type indicators, the indicator results obtained for each bucket can be integrated, and then the indicator result in the sample time experiment analysis dimension of the resampling bucket can be obtained. In addition, as shown in FIG. 3, the total quantity type indicators corresponding to the full-quantity bucket, i.e., all content operation records, can be determined, the ratio type indicators can be obtained based on the total quantity type indicators, and then the indicator result in the sample time experiment analysis dimension can be obtained. Figure 10 Figure 10

[0161] II. Test the effect of the experimental group

[0162] After the resampling bucket and indicator calculation operations are completed, the significance test of the effect difference is needed to obtain the evaluation conclusion of the current experiment effect. Let the indicator values of the experimental group and the control group be Y t and Y c . The null hypothesis H0 of the conventional experiment evaluation is that there is no difference between the experimental group and the control group, i.e., Y t -Y c ​​​​= 0. The inventors find that the scheme applies total quantity indicators and ratio indicators of various types in the advertising business, and each indicator has a large difference in the value range. If the difference is directly used, the display content of the evaluation conclusion of different indicators is very inconsistent. Therefore, the scheme sets the original hypothesis H0 and the corresponding alternative hypothesis H1 by using the indicator change rate, which can ensure that the difference values of different indicators are uniformly fixed in the range of [0, 1], so as to reduce the understanding cost of experimenters. The original hypothesis H0 and the corresponding alternative hypothesis H1 can be:

[0163]

[0164] For the significance judgment of the difference in experimental effect, the most concerned is the variance of the sample data. The embodiment of the present application completes the calculation of the variance based on the resampling method. After the user is divided into buckets, resampling processing is performed. At this time, the new sample refers to the calculated indicator improvement rate after resampling the buckets. The calculation of the indicator improvement rate can be realized by the following formula:

[0165]

[0166] B’ ti : Resampling result of the i th bucket of the experimental group

[0167] B’ ci : Resampling result of the i th bucket of the control group

[0168] After the above processing, the result set Y = {Y1, Y2, Y3,..., Y n} is obtained, each Y i is the indicator improvement rate of the experiment under the i th re-bucketing, and the next step can be mean calculation. The mean of the sample improvement rate can be calculated according to the following formula

[0169]

[0170] Combining the indicator improvement rate of the resampling sample and the mean value, the variance estimate value Var(Y) of the effect difference can be calculated by the following formula:

[0171]

[0172] After obtaining the variance estimate value of the experimental effect difference, according to the preset significance level a = 0.05, the significance interval of the effect difference, i.e. the 95% confidence interval, is calculated by the following formula:

[0173]

[0174] The real effect difference value has a 95% probability of falling in this interval, so whether the interval contains 0 can be used as a basis for judging whether the current experimental effect difference is significant, that is, if the significant interval does not contain 0, it means that the effect difference has a 95% probability of not being 0, and the difference is significant, otherwise it is not significant. At the same time, if the upper and lower bounds of the interval are both greater than 0, it means that the experimental strategy effect is significantly positive, otherwise the strategy effect is significantly negative. The experimental effect evaluation conclusion will be returned to the effect query API together with the observation results of the experimental indicators, and will be intuitively displayed to the experimenters through the Web front end. According to the query results, the user can judge whether the current strategy is truly effective.

[0175] The evaluation of the experimental effect focuses on the variance calculation in hypothesis testing. Although most A / B testing systems can correctly complete variance calculation and hypothesis testing and provide users with scientific experimental evaluation results, there are still many problems in specific implementation and use. In view of the above problems, the embodiment of the application adopts a combination of non-parametric testing and engineering framework, randomly divides the sample set into buckets, and forms a new sample set by discarding one bucket from the bucket set each time through the resampling method to complete the resampling operation. This greatly reduces the number of sample data records, reduces the storage and computing resources required for engineering implementation, and can save the experimental cross-multiplication data in different analysis dimensions, allowing experimenters to complete experimental effect evaluation in multiple dimension combinations. Based on this, the embodiment of the application has at least the following beneficial effects:

[0176] 1. Real-time experimental effect evaluation, data aggregation at the bucket level can greatly reduce the performance pressure of the system and reduce the long response time.

[0177] 2. Support for experimental effect evaluation in multiple dimension combinations, which can support the ability of the experiment to drill down to different analysis dimensions to complete effect evaluation, and can quickly determine the experimental evaluation results of the "time * sample number * experiment number * analysis dimension" multi-dimensional combination, with high experimental evaluation efficiency.

[0178] 3. Simple calculation logic, strong universality and expandability, the resampling method uses a general variance calculation formula that can be applied to different effect index calculation formulas, greatly simplifying the calculation logic and processing flow, supporting real-time processing and response to query requests, making the experimental effect evaluation framework have good flexibility and expandability.

[0179] 4. Relieve the pressure of data storage and computing resources.

[0180] Taking the application in the advertising experiment system as an example, after using the method of the application to replace the old parameter testing method and upgrading the experimental effect evaluation framework, it is concluded through comparison that the new scheme can bring significant optimization to various aspects of the system:

[0181]

[0182] From the comparison of the new and old evaluation frameworks, using the resampling method of the embodiments of the present application, that is, the new evaluation framework, the system only needs to process the data at the bucket level when evaluating the experimental effect, without processing the data at the user level one by one, and the data scale is reduced by about 90% (100 billion +→100 billion +), thereby reducing the computing resource investment by nearly 60%. At the same time, the new evaluation framework supports real-time completion of experimental effect evaluation under 20+ analysis dimensions. Due to the simplification of the computing logic, the experimental evaluation framework also has better scalability and processing performance, and the supported evaluation index type is expanded by about 150% (20+→50+). At present, the daily experimental evaluation request is increased by 3 times to nearly 30000 times, the average request response time is reduced by about 75% (40s+→10s+), and the new evaluation framework using the method of the embodiments of the present application can well ensure the processing performance while meeting the experimental demand which is increased by a large multiple, and improve the efficiency of the user to complete the experimental evaluation.

[0183] It should be understood that, although each step in the above flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowchart can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0184] Based on the same idea as the content pushing strategy processing method in the above embodiments, the present application also provides a content pushing strategy processing device, which can be used to execute the above content pushing strategy processing method. For the convenience of description, only the part related to the embodiments of the present application is shown in the structure diagram of the content pushing strategy processing device embodiment, and those skilled in the art can understand that the diagrammed structure does not constitute a limitation on the device, which can include more or fewer components than the diagrammed, or combine certain components, or different component arrangements.

[0185] In one embodiment, as shown in Figure 11 A content pushing strategy processing device 1100 is provided, which can be a part of a computer device in the form of a software module or a hardware module, or a combination of the two. The device specifically includes a record set obtaining module 1102, a record set dividing module 1104, a set sampling module 1106, an index value obtaining module 1108, and a test result obtaining module 1110, wherein:

[0186] The record set obtaining module 1102 is configured to obtain a reference content operation record set corresponding to the reference content pushing strategy and a test content operation record set corresponding to the test content pushing strategy.

[0187] The record set dividing module 1104 is configured to divide the reference content operation record set to obtain a plurality of reference record sub-sets and divide the test content operation record set to obtain a plurality of test record sub-sets.

[0188] The set sampling module 1106 is configured to sample the reference record sub-sets multiple times to obtain a sampling reference record sub-set corresponding to each sampling, and sample the test record sub-sets multiple times to obtain a sampling test record sub-set corresponding to each sampling.

[0189] The index value obtaining module 1108 is configured to take the sampling reference record sub-set as a sample calculation unit to obtain a reference operation index value corresponding to the sampling reference record sub-set, and take the sampling test record sub-set as a sample calculation unit to obtain a test operation index value corresponding to the sampling test record sub-set.

[0190] The test result obtaining module 1110 is configured to obtain a strategy test result corresponding to the test content pushing strategy based on a difference between the reference operation index value and the test operation index value.

[0191] In the content pushing strategy processing apparatus, the strategy test result corresponding to the test content pushing strategy is quickly obtained based on the difference between the reference operation index value and the test operation index value with a small data size, and the processing efficiency of the content pushing strategy is effectively improved.

[0192] In an embodiment, the set sampling module comprises: a first sub-set filtering sub-module configured to filter at least one reference record sub-set from the plurality of reference record sub-sets to obtain a filtered reference record sub-set and obtain a sampling reference record sub-set based on the filtered reference record sub-set at each sampling; and a second sub-set filtering sub-module configured to filter at least one test record sub-set from the plurality of test record sub-sets to obtain a filtered test record sub-set and obtain a sampling test record sub-set based on the filtered test record sub-set at each sampling.

[0193] In an embodiment, the first sub-set filtering sub-module is further configured to filter at least one reference content operation record from at least one reference record sub-set of the filtered reference record sub-set, combine the reference record sub-sets after filtering the reference content operation records to obtain the sampling reference record sub-set.

[0194] In an embodiment, the second subset filtering submodule is further configured to filter at least one test content operation record from each of the filtered test record subsets, combine the test record subsets after filtering the test content operation records, and obtain the sampling test record subset.

[0195] In an embodiment, the record set division module comprises: an attribute information acquisition submodule configured to acquire attribute information of a reference user corresponding to a reference content operation record; a division feature value determination submodule configured to obtain a division feature value corresponding to the reference content operation record based on the attribute information of the reference user; and a set division submodule configured to divide the reference content operation record set based on the division feature value corresponding to the reference content operation record, and obtain a plurality of reference record subsets.

[0196] In an embodiment, the division feature value determination submodule comprises: a feature mapping unit configured to perform feature mapping based on the attribute information of the reference user, and obtain a feature mapping value corresponding to the reference content operation record; a division quantity acquisition unit configured to acquire a division quantity corresponding to the reference content operation record set; and a division feature value determination unit configured to perform modulo calculation based on the feature mapping value and the division quantity, and obtain the division feature value corresponding to the reference content operation record.

[0197] In an embodiment, the test result acquisition module comprises: a promotion value determination submodule configured to compare the test operation indicator value matched with the sampling sequence with the reference operation indicator value, obtain an indicator promotion value of the test operation indicator value relative to the reference operation indicator value, and form an indicator promotion value set with each indicator promotion value; a statistical submodule configured to perform statistics on the indicator promotion value set, and obtain an indicator statistical value; a distribution range determination submodule configured to obtain a statistical distribution range corresponding to the indicator promotion value based on the difference between each indicator promotion value and the indicator statistical value; and a test result determination submodule configured to compare the statistical distribution range with a range threshold, and obtain a strategy test result corresponding to the test content pushing strategy based on the comparison result.

[0198] In an embodiment, the distribution range determination submodule comprises: a dispersion determination unit configured to obtain a promotion value dispersion corresponding to the indicator promotion value based on the difference between each indicator promotion value and the indicator statistical value; a test coefficient acquisition unit configured to acquire a test coefficient corresponding to a confidence probability threshold; an offset value determination unit configured to perform product calculation on the test coefficient and the promotion value dispersion, and obtain a test offset value; and a distribution range determination unit configured to obtain a statistical distribution range corresponding to the indicator promotion value according to the indicator statistical value and the test offset value.

[0199] In one embodiment, the test result determining submodule comprises: a threshold comparison unit configured to compare a minimum value corresponding to a statistical distribution range with the range threshold to obtain a comparison result; and a strategy pushing unit configured to, when the comparison result is that the minimum value is greater than the range threshold, determine that the test result of the test content pushing strategy is valid, take the test content pushing strategy as a target content pushing strategy, and push content using the target content pushing strategy.

[0200] In one embodiment, the record set obtaining module further comprises: a user determining submodule configured to determine a reference user and a test user from sample users; a pushing content determining submodule configured to obtain historical pushing content for a target pushing resource as reference pushing content matched with a reference content pushing strategy, and obtain current pushing content for the target pushing resource as test pushing content matched with a test content pushing strategy; a reference content pushing submodule configured to push the reference pushing content to a terminal of the reference user, and obtain reference content operation records in response to operations of the reference user on the reference pushing content; and a test content pushing submodule configured to push the test pushing content to a terminal of the test user, and obtain test content operation records in response to operations of the test user on the test pushing content.

[0201] In one embodiment, the device further comprises: a pushing strategy determining module configured to, when the strategy test result is valid, take the test content pushing strategy as a target content pushing strategy; the target content pushing strategy corresponds to target pushing content; a content feature obtaining module configured to obtain target content features corresponding to the target pushing content; a corresponding relationship obtaining module configured to obtain a corresponding relationship between content features and user features in the target content pushing strategy; and a pushing content pushing module configured to obtain target user features based on the corresponding relationship and the target content features, obtain a target user corresponding to the target user features, and push the target pushing content to a terminal corresponding to the target user.

[0202] In one embodiment, the record set obtaining module comprises: a target dimension obtaining submodule configured to obtain dimension selection information sent by a target terminal, and obtain a target dimension corresponding to the dimension selection information; a reference record set obtaining submodule configured to obtain content operation records matched with the target dimension from content operation records corresponding to a reference content pushing strategy to obtain a reference content operation record set; and a test record set obtaining submodule configured to obtain test content operation records matched with the target dimension from content operation records corresponding to a test content pushing strategy to obtain a test content operation record set.

[0203] The specific limitations of the content pushing strategy processing apparatus can refer to the limitations of the content pushing strategy processing method in the foregoing, and will not be described here. Each module in the content pushing strategy processing apparatus can be implemented by software, hardware, or a combination thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0204] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 12 The computer device includes a processor, a memory, and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store at least one of a reference content pushing strategy, a test content pushing strategy, a reference content operation record set, a test content operation record set, and the like. The network interface of the computer device is configured to communicate with a terminal outside through a network connection. The computer program is executed by the processor to implement a content pushing strategy processing method.

[0205] Those skilled in the art can understand that Figure 12 The structure shown in the above

[0206] In one embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in each method embodiment.

[0207] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in each method embodiment.

[0208] In one embodiment, a computer program product or a computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the steps in each method embodiment.

[0209] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0210] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0211] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A content push policy processing method, characterized by, The method comprises: obtaining a reference content operation record set corresponding to a reference content pushing strategy and a test content operation record set corresponding to a test content pushing strategy; obtaining attribute information of a reference user corresponding to the reference content operation record; obtaining a division feature value corresponding to the reference content operation record based on the attribute information of the reference user; dividing the reference content operation record set based on the division feature value corresponding to the reference content operation record to obtain a plurality of reference record sub-sets; dividing the test content operation record set to obtain a plurality of test record sub-sets; sampling the reference record sub-sets multiple times to obtain a sampling reference record sub-set corresponding to each sampling, and sampling the test record sub-sets multiple times to obtain a sampling test record sub-set corresponding to each sampling; calculating a reference operation index value corresponding to the sampling reference record sub-set as a sample calculation unit, and calculating a test operation index value corresponding to the sampling test record sub-set as a sample calculation unit; obtaining a strategy test result corresponding to the test content pushing strategy based on the difference between the reference operation index value and the test operation index value.

2. The method of claim 1, wherein, The sampling of the reference record sub-sets multiple times to obtain a sampling reference record sub-set corresponding to each sampling, and the sampling of the test record sub-sets multiple times to obtain a sampling test record sub-set corresponding to each sampling, comprises: at each sampling, filtering at least one reference record sub-set from the plurality of reference record sub-sets to obtain a filtered reference record sub-set, and obtaining a sampling reference record sub-set based on the filtered reference record sub-set; at each sampling, filtering at least one test record sub-set from the plurality of test record sub-sets to obtain a filtered test record sub-set, and obtaining a sampling test record sub-set based on the filtered test record sub-set.

3. The method of claim 2, wherein, The obtaining of the sampling reference record sub-set based on the filtered reference record sub-set comprises: filtering at least one reference content operation record from at least one reference record sub-set of the filtered reference record sub-set, and combining each reference record sub-set after filtering the reference content operation record to obtain the sampling reference record sub-set.

4. The method of claim 2, wherein, The obtaining of the sampling test record sub-set based on the filtered test record sub-set comprises: filtering at least one test content operation record from at least one test record sub-set of the filtered test record sub-set, and combining each test record sub-set after filtering the test content operation record to obtain the sampling test record sub-set.

5. The method of claim 1, wherein, The obtaining of the division feature value corresponding to the reference content operation record based on the attribute information of the reference user comprises: performing feature mapping based on the attribute information of the reference user to obtain a feature mapping value corresponding to the reference content operation record; obtaining a division number corresponding to the reference content operation record set; performing modulo calculation based on the feature mapping value and the division number to obtain the division feature value corresponding to the reference content operation record.

6. The method according to any one of claims 1 to 5, characterized in that, The strategy test result corresponding to the test content pushing strategy is obtained based on a difference between the reference operation index value and the test operation index value, and the strategy test result includes: The test operation index value matched with the sampling sequence is compared with the reference operation index value to obtain an index improvement value of the test operation index value relative to the reference operation index value, and each index improvement value forms an index improvement value set; The index improvement value set is counted to obtain an index statistical value; A statistical distribution range corresponding to the index improvement value is obtained based on a difference between each index improvement value and the index statistical value; The statistical distribution range is compared with a range threshold value, and a strategy test result corresponding to the test content pushing strategy is obtained based on a comparison result.

7. The method of claim 6, wherein, The statistical distribution range corresponding to the index improvement value is obtained based on a difference between each index improvement value and the index statistical value, and the statistical distribution range includes: A value dispersion of the index improvement value corresponding to the index improvement value is obtained based on a difference between each index improvement value and the index statistical value; A test coefficient corresponding to a confidence probability threshold value is obtained; A test deviation value is obtained by multiplying the test coefficient and the value dispersion; The statistical distribution range corresponding to the index improvement value is obtained according to the index statistical value and the test deviation value.

8. The method of claim 6, wherein, The statistical distribution range is compared with a range threshold value, and a strategy test result corresponding to the test content pushing strategy is obtained based on a comparison result, and the comparison includes: A comparison result is obtained by comparing a minimum value corresponding to the statistical distribution range with the range threshold value; When the comparison result is that the minimum value is greater than the range threshold value, it is determined that a test result of the test content pushing strategy is valid, and the test content pushing strategy is used as a target content pushing strategy to push content by using the target content pushing strategy.

9. The method according to any one of claims 1 to 5, characterized in that, A reference content operation record corresponding to a reference content operation record set is obtained, and a test content operation record corresponding to a test content operation record set is obtained, and the obtaining includes: Reference users and test users are determined from sample users; A historical pushing content for a target pushing resource is obtained as reference pushing content matched with a reference content pushing strategy, and a current pushing content for the target pushing resource is obtained as test pushing content matched with a test content pushing strategy; The reference pushing content is pushed to a terminal of the reference user, and a reference content operation record is obtained in response to an operation of the reference user on the reference pushing content; The test pushing content is pushed to a terminal of the test user, and a test content operation record is obtained in response to an operation of the test user on the test pushing content.

10. The method according to any one of claims 1 to 5, characterized in that, The method further includes: When the strategy test result is valid, the test content pushing strategy is used as a target content pushing strategy, and a pushing content corresponding to the target content pushing strategy is a target pushing content; A target content feature corresponding to the target pushing content is obtained; A corresponding relationship between a content feature and a feature of a user in the target content pushing strategy is obtained; Based on the correspondence and the target content feature, a feature of the target user is acquired, a target user corresponding to the feature of the target user is acquired, and the target push content is pushed to a terminal corresponding to the target user.

11. The method according to any one of claims 1 to 5, characterized in that, The reference content operation record set corresponding to the reference content push strategy and the test content operation record set corresponding to the test content push strategy include: Dimension selection information sent by a target terminal is acquired, a target dimension corresponding to the dimension selection information is acquired; In the content operation record corresponding to the reference content push strategy, a content operation record matching the target dimension is acquired, and a reference content operation record set is obtained; In the content operation record corresponding to the test content push strategy, a test content operation record matching the target dimension is acquired, and a test content operation record set is obtained.

12. A content push policy processing apparatus characterized by comprising: The device includes: A record set acquisition module is configured to acquire a reference content operation record set corresponding to a reference content push strategy and a test content operation record set corresponding to a test content push strategy; A record set division module includes an attribute information acquisition submodule, a division feature value determination submodule, and a set division submodule; The attribute information acquisition submodule is configured to acquire attribute information of a reference user corresponding to the reference content operation record; The division feature value determination submodule is configured to obtain a division feature value corresponding to the reference content operation record based on the attribute information of the reference user; The set division submodule is configured to divide the reference content operation record set based on the division feature value corresponding to the reference content operation record, and obtain a plurality of reference record sub-sets; The set division submodule is also configured to divide the test content operation record set, and obtain a plurality of test record sub-sets; A set sampling module is configured to sample the reference record sub-sets multiple times to obtain a sampling reference record sub-set corresponding to each sampling, and sample the test record sub-sets multiple times to obtain a sampling test record sub-set corresponding to each sampling; An index value acquisition module is configured to take the sampling reference record sub-set as a sample calculation unit to acquire a reference operation index value corresponding to the sampling reference record sub-set, and take the sampling test record sub-set as a sample calculation unit to acquire a test operation index value corresponding to the sampling test record sub-set; A test result acquisition module is configured to obtain a strategy test result corresponding to the test content push strategy based on a difference between the reference operation index value and the test operation index value.

13. The content push policy processing apparatus according to claim 12, characterized by The set sampling module includes: A first sub-set filtering submodule is configured to filter at least one reference record sub-set from a plurality of reference record sub-sets at each sampling to obtain a filtered reference record sub-set, and obtain a sampling reference record sub-set based on the filtered reference record sub-set; A second sub-set filtering submodule is configured to filter at least one test record sub-set from a plurality of test record sub-sets at each sampling to obtain a filtered test record sub-set, and obtain a sampling test record sub-set based on the filtered test record sub-set.

14. The content push policy processing apparatus according to claim 13, characterized by The first sub-set filtering submodule is further configured to filter at least one reference content operation record from at least one reference record sub-set of the filtered reference record sub-sets, combine each reference record sub-set after filtering the reference content operation record to obtain a sampling reference record sub-set.

15. The content push policy processing apparatus of claim 13, wherein, The second sub-set filtering submodule is further configured to filter at least one test content operation record from at least one test record sub-set of the filtered test record sub-sets, combine each test record sub-set after filtering the test content operation record to obtain a sampling test record sub-set.

16. The content push policy processing apparatus of claim 12, wherein, The division feature value determination submodule comprises: a feature mapping unit configured to perform feature mapping based on attribute information of the reference user to obtain a feature mapping value corresponding to the reference content operation record; a division quantity acquisition unit configured to acquire a division quantity corresponding to the reference content operation record set; a division feature value determination unit configured to perform modulo calculation based on the feature mapping value and the division quantity to obtain a division feature value corresponding to the reference content operation record.

17. The content push policy handling apparatus according to any one of claims 12 to 16, characterized by, The test result acquisition module comprises: a promotion value determination submodule configured to compare the test operation index value matched with the sampling sequence with the reference operation index value to obtain an index promotion value of the test operation index value relative to the reference operation index value, and each index promotion value constitutes an index promotion value set; a statistical submodule configured to perform statistics on the index promotion value set to obtain an index statistical value; a distribution range determination submodule configured to obtain a statistical distribution range corresponding to the index promotion value based on a difference between each index promotion value and the index statistical value; a test result determination submodule configured to compare the statistical distribution range with a range threshold value, and obtain a strategy test result corresponding to the test content push strategy based on a comparison result.

18. The content push policy processing apparatus of claim 17, wherein, The distribution range determination submodule comprises: a dispersion determination unit configured to obtain a promotion value dispersion corresponding to the index promotion value based on a difference between each index promotion value and the index statistical value; a test coefficient acquisition unit configured to acquire a test coefficient corresponding to a confidence probability threshold value; an offset value determination unit configured to perform product calculation on the test coefficient and the promotion value dispersion to obtain a test offset value; a distribution range determination unit configured to obtain a statistical distribution range corresponding to the index promotion value according to the index statistical value and the test offset value.

19. The content push policy processing apparatus of claim 17, wherein, The test result determination submodule comprises: a threshold comparison unit configured to compare a minimum value corresponding to the statistical distribution range with the range threshold value to obtain a comparison result; a strategy push unit configured to determine a test result of the test content push strategy as valid when the comparison result is that the minimum value is greater than the range threshold value, and use the test content push strategy as a target content push strategy to perform content push by using the target content push strategy.

20. The content push policy handling apparatus according to any one of claims 12 to 16, characterized by, The record set acquisition module comprises: a user determination submodule configured to determine a reference user and a test user from sample users; The push content determination submodule is configured to acquire historical push content for a target push resource as reference push content for reference content push strategy matching, and acquire current push content for the target push resource as test push content for test content push strategy matching; The reference content push submodule is configured to push the reference push content to a terminal of the reference user, and obtain reference content operation records in response to operations of the reference user on the reference push content; The test content push submodule is configured to push the test push content to a terminal of the test user, and obtain test content operation records in response to operations of the test user on the test push content.

21. The content push policy handling apparatus according to any one of claims 12 to 16, characterized by, The apparatus further includes: The push strategy determination module is configured to, when the strategy test result is valid, take the test content push strategy as a target content push strategy; the target content push strategy corresponds to target push content; The content feature acquisition module is configured to acquire target content features corresponding to the target push content; The corresponding relationship acquisition module is configured to acquire a corresponding relationship between content features and features of users in the target content push strategy; The push content push module is configured to acquire features of target users based on the corresponding relationship and the target content features, acquire target users corresponding to the features of the target users, and push the target push content to terminals corresponding to the target users.

22. The content push policy handling apparatus according to any one of claims 12 to 16, characterized by The record set acquisition module includes: The target dimension acquisition submodule is configured to acquire dimension selection information sent by a target terminal, and acquire a target dimension corresponding to the dimension selection information; The reference record set acquisition submodule is configured to acquire content operation records matching the target dimension from content operation records corresponding to a reference content push strategy, and obtain a reference content operation record set; The test record set acquisition submodule is configured to acquire test content operation records matching the target dimension from content operation records corresponding to a test content push strategy, and obtain a test content operation record set. 23.A computer device, comprising a memory and a processor, wherein the memory stores a computer program. The processor executes the computer program to implement the method in any one of claims 1 to 11.

24. A computer readable storage medium storing a computer program, wherein the computer program comprises instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-23. The computer program is executed by the processor to implement the method in any one of claims 1 to 11.

25. A computer program product comprising computer instructions, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 11.

Images