Recommendation information processing method, recommendation information ranking method and device

Abstract

The present disclosure provides a recommendation information processing method, a recommendation information ranking method and device. The recommendation information processing method comprises: determining a plurality of attributes as a plurality of candidate aggregation dimensions from a set of attributes associated with target recommendation information by solving a target function of a dimension optimization model, and determining a priority of each candidate aggregation dimension; selecting a reference aggregation dimension of the target recommendation information from the plurality of candidate aggregation dimensions based on at least the priority of each candidate aggregation dimension; obtaining a total estimated conversion amount and a total actual conversion amount of a reference recommendation information set corresponding to the target recommendation information before a current time in a current delivery period by aggregating an estimated conversion amount and an actual conversion amount of a reference recommendation information in the reference recommendation information set under the reference aggregation dimension; determining a calibration parameter based on at least the total estimated conversion amount and the total actual conversion amount; and calibrating an estimated conversion rate of the target recommendation information using the calibration parameter.

Description

Technical Field

[0001] This disclosure relates to the field of multimedia, and more specifically, to a method for processing recommendation information, a method for ranking recommendation information, and an apparatus. Background Technology

[0002] In the multimedia field, multimedia platforms frequently recommend information to target groups through various means, such as advertisements and news. Taking advertising as an example, advertising conversion rate prediction plays a crucial role in achieving precise ad targeting and evaluating ad performance. While the predicted conversion rate (PCVR) obtained through current conversion rate prediction models has good overall performance, its accuracy is not high for specific industries, audiences, or content. Therefore, after obtaining the predicted conversion rate, it is often necessary to calibrate the PCVR by combining various attributes such as industry, audience, content, and brand. For example, by aggregating the delivery data of related advertisements with common attributes, the PCVR of an advertisement can be calibrated specifically for a particular industry to obtain the most accurate PCVR value. How to select the best attribute from the many attributes of an advertisement as the aggregation dimension to obtain aggregated delivery data of related advertisements, and how to use the determined aggregation dimension to achieve PCVR calibration and obtain the most accurate PCVR value, remain to be solved. Summary of the Invention

[0003] To address the aforementioned problems, this disclosure provides a method for processing recommendation information, a method for ranking recommendation information, an apparatus and device, a computer-readable storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, a method for processing recommendation information is provided, comprising: determining multiple attributes as multiple candidate aggregation dimensions from a set of attributes associated with the target recommendation information based on historical delivery data of a predetermined set of recommendation information containing target recommendation information within a historical delivery period, and determining the priority of each of the multiple candidate aggregation dimensions; selecting a reference aggregation dimension for the target recommendation information from the multiple candidate aggregation dimensions based at least on the priority of each of the multiple candidate aggregation dimensions; obtaining the total estimated conversion volume and total actual conversion volume of the reference recommendation information set before the current time in the current delivery period by aggregating the estimated conversion volume and actual conversion volume of the reference recommendation information set corresponding to the target recommendation information under the reference aggregation dimension; determining a calibration parameter based at least on the total estimated conversion volume and the total actual conversion volume of the reference recommendation information set; and calibrating the estimated conversion rate corresponding to a click on the target recommendation information at the current time using the calibration parameter.

[0005] According to an example of an embodiment of this disclosure, a dimensional optimization model is established based on historical delivery data of a predetermined set of recommendation information containing target recommendation information within a historical delivery period. By solving the objective function of the dimensional optimization model, multiple attributes are determined from a set of attributes associated with the target recommendation information as multiple candidate aggregation dimensions, and the priority of each candidate aggregation dimension among the multiple candidate aggregation dimensions is determined.

[0006] According to an example of an embodiment of this disclosure, the objective function of the dimensional optimization model is the sum of the calibration deviations of the historical estimated conversion rates of the predetermined recommendation information in the predetermined recommendation information set within the historical delivery period.

[0007] According to an example of an embodiment of this disclosure, the process of determining multiple attributes as multiple candidate aggregation dimensions from a set of attributes associated with target recommendation information by solving the objective function of the dimensional optimization model, and determining the priority of each candidate aggregation dimension, includes: randomly selecting multiple attributes as multiple initial aggregation dimensions from the set of attributes associated with the target recommendation information, each of the multiple initial aggregation dimensions having a set priority; using the multiple initial aggregation dimensions and the set priority, calibrating the historical estimated conversion rate corresponding to each historical click of each predetermined recommendation information in the predetermined recommendation information set within a historical delivery period; calculating the calibration deviation between the calibrated historical estimated conversion volume and the historical actual conversion volume of each predetermined recommendation information, and summing the calibration deviations of the estimated conversion volumes of the predetermined recommendation information in the predetermined recommendation information set within a historical delivery period as the objective function; and determining the multiple initial aggregation dimensions that minimize the objective function as the multiple candidate aggregation dimensions, and determining the set priority of each of the multiple initial aggregation dimensions that minimizes the objective function as the priority of the corresponding candidate aggregation dimension.

[0008] According to an example of an embodiment of this disclosure, calibrating the historical estimated conversion rate corresponding to each historical click of each predetermined recommendation information in the predetermined recommendation information set using the plurality of initial aggregation dimensions and the set priority includes: for each predetermined recommendation information in the predetermined recommendation information set: selecting an initial reference aggregation dimension from the plurality of initial aggregation dimensions based at least on the set priority of each initial aggregation dimension; determining historical calibration parameters based at least on the historical total estimated conversion volume and historical total actual conversion volume of the reference recommendation information set under the initial reference aggregation dimension corresponding to the predetermined recommendation information within the historical campaign period; and calibrating the historical estimated conversion rate corresponding to each historical click of the predetermined recommendation information within the historical campaign period using the historical calibration parameters.

[0009] According to an example of an embodiment of this disclosure, selecting a reference aggregation dimension for the target recommendation information from the plurality of candidate aggregation dimensions, based at least on the priority of each of the plurality of candidate aggregation dimensions, includes: obtaining the total delivery cost of the set of recommendation information corresponding to the target recommendation information under each of the plurality of candidate aggregation dimensions; and selecting the candidate aggregation dimension with the highest priority among the plurality of candidate aggregation dimensions, where the total delivery cost of the set of recommendation information is greater than a first predetermined threshold, as the reference aggregation dimension for the target recommendation information.

[0010] According to an example of an embodiment of this disclosure, before determining calibration parameters based at least on the total estimated conversion volume and the total actual conversion volume of the reference recommendation information set, the recommendation information processing method further includes: determining, based on predetermined rules, whether the target recommendation information is in the initial stage or the mature stage of the current campaign period at the current moment; wherein, when the target recommendation information is in the initial stage, the recommendation information processing method further includes: obtaining the total estimated click volume and the total actual click volume of the reference recommendation information set before the current moment in the current campaign period; and wherein, when the target recommendation information is in the mature stage, the recommendation information processing method further includes: obtaining the estimated conversion volume, actual conversion volume, estimated click volume, and actual click volume of the target recommendation information before the current moment in the current campaign period.

[0011] According to an example of an embodiment of this disclosure, determining whether the target recommendation information is in the initial stage or the mature stage of the current campaign period at the current time based on predetermined rules includes: determining the current campaign consumption and current conversion volume generated by the target recommendation information before the current time in the current campaign period; when the current campaign consumption is less than or equal to a second predetermined threshold and the current conversion volume is less than or equal to a third predetermined threshold, determining that the target recommendation information is in the initial stage of the campaign; otherwise, determining that the target recommendation information is in the mature stage of the campaign.

[0012] According to an example of an embodiment of this disclosure, when the target recommendation information is in the initial stage of delivery, determining the calibration parameters based at least on the total estimated conversion volume and the total actual conversion volume of the reference recommendation information set includes: determining the calibration parameters based at least on the total estimated conversion volume, the total actual conversion volume, the total estimated click volume, and the total actual click volume of the reference recommendation information set.

[0013] According to one example of an embodiment of this disclosure, determining the calibration parameter based at least on the total estimated conversion, total actual conversion, total estimated clicks, and total actual clicks of the reference recommendation information set includes: obtaining the current campaign cost generated by the target recommendation information before the current time in the current campaign period, and determining an expansion coefficient based on the current campaign cost; and determining the calibration parameter based on the expansion coefficient and the total estimated conversion, total actual conversion, total clicks, and total actual clicks of the reference recommendation information set.

[0014] According to an example of an embodiment of this disclosure, when the target recommendation information is in the mature stage of deployment, determining the calibration parameters based at least on the total estimated conversion volume and the total actual conversion volume of the reference recommendation information set includes: determining the calibration parameters based at least on the total estimated conversion volume and the total actual conversion volume of the reference recommendation information set, and the estimated conversion volume, actual conversion volume, estimated click volume, and actual click volume of the target recommendation information before the current time in the current deployment period.

[0015] According to an example of an embodiment of this disclosure, determining the calibration parameter based at least on the total estimated conversion rate and total actual conversion rate of the reference recommendation information set, and the estimated conversion rate, actual conversion rate, estimated click-through rate, and actual click-through rate of the target recommendation information up to the current time in the current campaign period includes: obtaining the historical actual conversion rate of the target recommendation information in a historical campaign period, and determining a smoothing coefficient based on the historical actual conversion rate of the target recommendation information and the length of time up to the current time in the current campaign period; and determining the calibration parameter based on the smoothing coefficient, the total estimated conversion rate and total actual conversion rate of the reference recommendation information set, and the estimated conversion rate, actual conversion rate, estimated click-through rate, and actual click-through rate of the target recommendation information up to the current time in the current campaign period.

[0016] According to an example of an embodiment of this disclosure, the attribute set includes at least a subset of attribution attributes, a subset of audience attributes, and a subset of content attributes of the target recommendation information, and determining multiple attributes as multiple candidate aggregation dimensions from the attribute set associated with the target recommendation information includes: determining at least one attribute from the subset of attribution attributes, the subset of audience attributes, and the subset of content attributes as a candidate aggregation dimension among the multiple candidate aggregation dimensions.

[0017] According to another aspect of the embodiments of this disclosure, a method for ranking recommendation information is provided, comprising: obtaining a calibrated estimated conversion rate for each of a plurality of recommendation information to be delivered; calculating an estimated revenue for each of the plurality of recommendation information to be delivered based on the calibrated estimated conversion rate; and ranking the plurality of recommendation information to be delivered based on the estimated revenue, and delivering the plurality of recommendation information to be delivered sequentially according to the ranking result, wherein obtaining the calibrated estimated conversion rate for each of the plurality of recommendation information to be delivered includes: calibrating the estimated conversion rate of each of the plurality of recommendation information to be delivered using the recommendation information processing method described above, to obtain the calibrated estimated conversion rate of each of the plurality of recommendation information to be delivered.

[0018] According to another aspect of the present disclosure, a recommendation information processing apparatus is provided, comprising: a dimension determination unit configured to determine multiple attributes as multiple candidate aggregation dimensions from a set of attributes associated with the target recommendation information based on historical delivery data of a predetermined set of recommendation information containing target recommendation information within a historical delivery period, and to determine the priority of each of the multiple candidate aggregation dimensions; a selection unit configured to select a reference aggregation dimension from the multiple candidate aggregation dimensions based at least on the priority of each of the multiple candidate aggregation dimensions; an aggregation unit configured to obtain the total estimated conversion volume and total actual conversion volume of the reference recommendation information set before the current time in the current delivery period by aggregating the estimated conversion volume and actual conversion volume of the reference recommendation information set corresponding to the target recommendation information under the reference aggregation dimension; a calibration parameter determination unit configured to determine calibration parameters based at least on the total estimated conversion volume and the total actual conversion volume of the reference recommendation information set; and a calibration unit configured to calibrate the estimated conversion rate corresponding to a click on the target recommendation information at the current time using the calibration parameters.

[0019] According to an example of an embodiment of this disclosure, a dimensional optimization model is established based on historical delivery data of a predetermined set of recommendation information containing target recommendation information within a historical delivery period. By solving the objective function of the dimensional optimization model, multiple attributes are determined from a set of attributes associated with the target recommendation information as multiple candidate aggregation dimensions, and the priority of each candidate aggregation dimension among the multiple candidate aggregation dimensions is determined.

[0020] According to an example of an embodiment of this disclosure, the objective function of the dimensional optimization model is the sum of the calibration deviations of the historical estimated conversion rates of the predetermined recommendation information in the predetermined recommendation information set within the historical delivery period.

[0021] According to an example of an embodiment of this disclosure, the dimension determination unit is configured to: randomly select multiple attributes from a set of attributes associated with the target recommendation information as multiple initial aggregation dimensions, each of the multiple initial aggregation dimensions having a set priority; calibrate the historical estimated conversion rate corresponding to each historical click of each predetermined recommendation information in the predetermined recommendation information set within a historical delivery period using the multiple initial aggregation dimensions and the set priority; calculate the calibration deviation between the calibrated historical estimated conversion volume and the historical actual conversion volume of each predetermined recommendation information, and calculate the cumulative sum of the calibration deviations of the estimated conversion volumes of the predetermined recommendation information in the predetermined recommendation information set within a historical delivery period as the objective function; and determine the multiple initial aggregation dimensions that minimize the objective function as the multiple candidate aggregation dimensions, and determine the set priority of each of the multiple initial aggregation dimensions that minimizes the objective function as the priority of the corresponding candidate aggregation dimension.

[0022] According to an example of an embodiment of this disclosure, the dimension determination unit is further configured to: for each predetermined recommendation information in the predetermined recommendation information set: select an initial reference aggregation dimension from the plurality of initial aggregation dimensions based at least on the set priority of each initial aggregation dimension; determine historical calibration parameters based at least on the historical total estimated conversion volume and historical total actual conversion volume of the reference recommendation information set under the initial reference aggregation dimension corresponding to the predetermined recommendation information within the historical campaign period; and calibrate the historical estimated conversion rate corresponding to each historical click of the predetermined recommendation information within the historical campaign period using the historical calibration parameters.

[0023] According to an example of an embodiment of this disclosure, the selection unit is configured to: obtain the total delivery cost of the set of recommendation information corresponding to the target recommendation information under each of the plurality of candidate aggregation dimensions; and select the candidate aggregation dimension with the highest priority among the plurality of candidate aggregation dimensions whose total delivery cost of the set of recommendation information is greater than a first predetermined threshold as the reference aggregation dimension of the target recommendation information.

[0024] According to an example of an embodiment of this disclosure, the recommendation information processing device further includes a placement stage determination unit, which is configured to: determine, based on predetermined rules, whether the target recommendation information is in the initial placement stage or the mature placement stage of the current placement period at the current moment; wherein, when the target recommendation information is in the initial placement stage, the aggregation unit is further configured to: obtain the total estimated clicks and total actual clicks of the reference recommendation information set before the current moment in the current placement period; and wherein, when the target recommendation information is in the mature placement stage, the aggregation unit is further configured to: obtain the estimated conversion rate, actual conversion rate, estimated click rate, and actual click rate of the target recommendation information before the current moment in the current placement period.

[0025] According to an example of an embodiment of this disclosure, the delivery stage determination unit is further configured to: determine the current delivery consumption and current conversion amount generated by the target recommendation information before the current time in the current delivery cycle; when the current delivery consumption is less than or equal to a second predetermined threshold and the current conversion amount is less than or equal to a third predetermined threshold, determine that the target recommendation information is in the initial delivery stage; otherwise, determine that the target recommendation information is in the mature delivery stage.

[0026] According to an example of an embodiment of this disclosure, when the target recommendation information is in the initial stage of delivery, the calibration parameter determination unit is configured to determine the calibration parameters based at least on the total estimated conversion, total actual conversion, total estimated clicks, and total actual clicks of the reference recommendation information set.

[0027] According to an example of an embodiment of this disclosure, when the target recommendation information is in the initial stage of delivery, the calibration parameter determination unit is further configured to: obtain the current delivery consumption generated by the target recommendation information before the current time in the current delivery cycle, and determine an expansion coefficient based on the current delivery consumption; and determine the calibration parameter based on the expansion coefficient and the total estimated conversion, total actual conversion, total estimated clicks, and total actual clicks of the reference recommendation information set.

[0028] According to an example of an embodiment of this disclosure, when the target recommendation information is in the mature stage of deployment, the calibration parameter determination unit is configured to: determine the calibration parameters based at least on the total estimated conversion volume and total actual conversion volume of the reference recommendation information set, and the estimated conversion volume, actual conversion volume, estimated click volume, and actual click volume of the target recommendation information before the current time in the current deployment period.

[0029] According to an example of an embodiment of this disclosure, when the target recommendation information is in the mature stage of deployment, the calibration parameter determination unit is further configured to: obtain the historical actual conversion volume of the target recommendation information within a historical deployment period, and determine a smoothing coefficient based on the historical actual conversion volume of the target recommendation information and the time length up to the current moment in the current deployment period; and determine the calibration parameter based on the smoothing coefficient, the total estimated conversion volume and total actual conversion volume of the reference recommendation information set, and the estimated conversion volume, actual conversion volume, estimated click volume, and actual click volume of the target recommendation information before the current moment in the current deployment period.

[0030] According to an example of an embodiment of this disclosure, the attribute set includes at least a subset of attribution attributes, a subset of audience attributes, and a subset of content attributes of the target recommendation information, and the dimension determination unit is further configured to: determine at least one attribute from the subset of attribution attributes, the subset of audience attributes, and the subset of content attributes as a candidate aggregation dimension among the plurality of candidate aggregation dimensions.

[0031] According to another aspect of the present disclosure, a recommendation information sorting apparatus is provided, comprising: an acquisition unit configured to acquire a calibrated estimated conversion rate for each of a plurality of recommendation information to be delivered; a revenue estimation unit configured to calculate an estimated revenue for each of the plurality of recommendation information to be delivered based on the calibrated estimated conversion rate; and a sorting unit configured to sort the plurality of recommendation information to be delivered based on the estimated revenue of each of the plurality of recommendation information to be delivered, and to deliver the plurality of recommendation information to be delivered sequentially according to the sorting result, wherein the acquisition unit is further configured to: calibrate the estimated conversion rate of each of the plurality of recommendation information to be delivered using the recommendation information processing method described above, so as to acquire the calibrated estimated conversion rate of each of the plurality of recommendation information to be delivered.

[0032] According to another aspect of the present disclosure, a recommendation information processing apparatus is provided, comprising: one or more processors; and one or more memories, wherein the memories store computer-readable code that, when executed by the one or more processors, causes the one or more processors to perform the methods as described in any of the foregoing aspects of the present disclosure.

[0033] According to another aspect of the present disclosure, a computer-readable storage medium is provided that stores computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in any of the foregoing aspects of the present disclosure.

[0034] According to another aspect of the present disclosure, a computer program product is provided, which includes computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any of the foregoing aspects of the present disclosure.

[0035] By utilizing the recommendation information processing method, recommendation information ranking method, apparatus, device, computer-readable storage medium, and computer program product according to the above aspects of this disclosure, it is possible to determine multiple candidate aggregation dimensions that can achieve optimal PCVR calibration and the priority of each candidate aggregation dimension by solving the objective function of the dimensional optimization model for a predetermined set of recommendation information containing target recommendation information within a historical delivery period. This allows for precise PCVR calibration of the target recommendation information using multiple candidate aggregation dimensions and their priorities, significantly improving the accuracy of PCVR calibration. Furthermore, targeted calibration can be implemented by analyzing the characteristics of the estimated conversion rate of recommendation information for specific industries, specific audiences, and specific content, enabling more accurate recommendation information delivery, cost control, and prediction of delivery effects. Additionally, the delivery revenue of recommendation information can be estimated based on the calibrated estimated conversion rate, and multiple recommendations to be delivered can be ranked based on the estimated revenue, allowing priority delivery of recommendations with higher estimated revenue to maximize returns. Attached Figure Description

[0036] The above and other objects, features, and advantages of the present disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0037] Figure 1 A schematic diagram illustrating an application scenario according to an embodiment of this disclosure is shown;

[0038] Figure 2 A flowchart of a recommendation information processing method according to an embodiment of the present disclosure is shown;

[0039] Figure 3 A flowchart illustrating an example of solving a dimensionality optimization model according to an embodiment of this disclosure is shown;

[0040] Figure 4 The statistical mean of the calibrated estimated conversion rate deviation of a predetermined set of ads under an example of an embodiment of the present disclosure is shown.

[0041] Figure 5 The distribution ratio of calibrated estimated conversion rate deviations for a predetermined set of ads under an example of an embodiment of this disclosure is shown.

[0042] Figure 6 The statistical mean of the calibrated estimated conversion rate deviation of a predetermined set of advertisements under the advertiser dimension is shown in an example according to an embodiment of the present disclosure;

[0043] Figure 7 The distribution of calibrated estimated conversion rate deviations for a predetermined set of ads under the advertiser dimension, as shown in an example according to an embodiment of this disclosure, is illustrated.

[0044] Figure 8 A flowchart of a recommendation information sorting method according to an embodiment of the present disclosure is shown;

[0045] Figure 9 A schematic diagram of the structure of a recommendation information processing apparatus according to an embodiment of the present disclosure is shown;

[0046] Figure 10 A schematic diagram of the structure of a recommendation information sorting apparatus according to an embodiment of the present disclosure is shown;

[0047] Figure 11 A schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0048] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0049] In the embodiments of this disclosure, the recommendation information may be information pushed to the terminal device for display in the form of images, text, videos, or any combination thereof. For example, the recommendation information may be any content such as advertisements, recommended news, article / video promotions, etc., and this disclosure does not impose specific limitations on this. In the following embodiments of this disclosure, advertisements will be mainly used as examples of recommendation information for description, but those skilled in the art should understand that the various parts described using advertisements as examples are also applicable to other recommendation information.

[0050] The following describes various definitions that may be used in embodiments of this disclosure, using advertisements as examples of recommendation information. In embodiments of this disclosure, an advertiser generally refers to the party that funds the placement of advertisements, an advertising platform refers to the party that uses its own platform or technology to help advertisers with advertisement placement, analysis, prediction, etc., and the audience refers to the user group that views the advertisements. After an advertisement is placed through an advertising platform, it typically goes through the following process: First, the user sees the advertisement; this process is called exposure. After exposure, interested users may click to browse the products contained in the advertisement, for example, by clicking on a product link in the advertisement to jump to a product page for browsing; this behavior is called a click. After jumping to the product page by clicking, the user may purchase the product on the product page, or download and install the application on the product page, etc.; this behavior is called a conversion. In this disclosure, the ratio of ad clicks to ad exposures can be called the click-through rate (CTR), and the ratio of ad conversions to ad clicks can be called the conversion rate (CVR).

[0051] To achieve precise ad targeting and evaluate ad performance, models are typically used to predict conversion rates after ad placement, yielding the Predicted Conversion Rate (PCVR). With the development of deep learning, most current models for predicting ad conversion rates are based on neural networks, such as Deep Crossing models, Product Neural Networks (PNNs), and Factorization Machines (FMs). For example, each click on an ad corresponds to a certain probability of conversion, i.e., a specific conversion rate. PNNs, FMs, and other models can be used to predict the predicted conversion rate for each click. However, while the overall performance of these models' predicted conversion rates is good, their accuracy is not high for specific industries, audiences, or content. Therefore, targeted calibration of the predicted conversion rates is necessary.

[0052] Typically, PCVR calibration can be performed using the already generated delivery data of the target ad. However, in the initial delivery phase of a target ad, sufficient delivery data for PCVR calibration is often not yet available. In this case, useful information can be obtained by aggregating the delivery data of ads with certain similar attributes to the target ad (referred to as related ads) to calibrate the target ad's PCVR. For example, the delivery data of related ads with the same advertiser as the target ad may have a certain degree of similarity to the target ad. This related ad delivery data can then be used to calibrate the target ad's PCVR. That is, using the same advertiser as the aggregation dimension, the delivery data of all ads under that advertiser are aggregated, and the aggregated data is used to calibrate the target ad's PCVR.

[0053] However, targeted ads possess a vast set of attributes, including affiliation attributes such as the advertiser, their group, company, industry, and region; audience attributes such as gender, age, region, occupation, and behavioral interests; and content attributes such as product name, product type, industry type, title, and image. Selecting the optimal aggregation dimension for PCVR calibration from this extensive set of attributes is crucial. Furthermore, after obtaining accurate predicted conversion rates through calibration, the predicted revenue for each ad needs to be calculated to determine the ad delivery order. To address these issues, this disclosure provides a recommendation information processing method, a recommendation information ranking method, and corresponding apparatus and equipment.

[0054] First refer to Figure 1 The present disclosure describes application scenarios of the recommendation information processing method, recommendation information sorting method, and corresponding apparatus according to embodiments of the present disclosure. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and multiple terminals 120 are schematically illustrated. Figure 1 As shown, recommendation information, such as advertisements, can be displayed on multiple terminals 120 via server 110. When users on multiple terminals 120 browse the advertisements, various data such as exposure, clicks, and conversions may be generated; these are collectively referred to as delivery data. The recommendation information processing method, recommendation information sorting method, and corresponding apparatus according to embodiments of this disclosure can be mounted on server 110 to calibrate the estimated conversion rate of the recommendation information delivered to multiple terminals 120, and determine the order of the next delivery of each recommendation message based on the calibrated estimated conversion rate.

[0055] The server 110 here can be a standalone server for analyzing and processing recommendation information, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, location services, and big data and artificial intelligence platforms. This embodiment of the disclosure does not impose specific limitations on this. Each of the multiple terminals 120 can be a fixed terminal such as a desktop computer, a mobile terminal such as a smartphone, tablet computer, portable computer, handheld device, personal digital assistant, smart wearable device, or any combination thereof. This embodiment of the disclosure does not impose specific limitations on this.

[0056] The following reference Figure 2 A method for processing recommendation information according to embodiments of this disclosure is described. Figure 2A flowchart of a recommendation information processing method 200 according to an embodiment of the present disclosure is shown. Figure 2 As shown, in step S210, based on the dimensional optimization model established using historical delivery data within a historical delivery period for a predetermined set of recommendation information containing target recommendation information, multiple attributes are determined as multiple candidate aggregation dimensions from the attribute set associated with the target recommendation information by solving the objective function of the dimensional optimization model, and the priority of each candidate aggregation dimension is determined. The target recommendation information can be any recommendation information to be calibrated for PCVR, such as advertisements, recommended information, article / video promotions, etc., and this embodiment does not impose specific limitations on it.

[0057] According to an example of an embodiment of this disclosure, the set of attributes associated with the target recommendation information may include at least an attribution attribute subset, an audience attribute subset, and a content attribute subset. As their names suggest, the attribution attribute subset includes multiple attributes related to the attribution of the recommendation information, the audience attribute subset includes multiple attributes related to the audience of the recommendation information, and the content attribute subset includes multiple attributes related to the content of the recommendation information. The following description uses an advertisement as an example to illustrate each attribute subset.

[0058] The attribution attribute subset may include attributes such as advertiser, advertiser's group, advertiser's company, industry of the advertisement, sub-industry of the advertisement, category of the advertisement, region of the advertisement, operation project group of the advertisement, operation team of the advertisement, size level of the advertiser, size level of the advertiser's group, size level of the advertiser's company, etc.

[0059] Audience attribute subsets may include, for example, basic advertising targeting attributes such as gender, age, region, occupation, education level, and income level of the target audience; behavioral interest targeting attributes such as user app installation behavior, ad interaction behavior, e-commerce browsing behavior, search click behavior, news browsing behavior, and industry categories of interest; and advertiser-customized attributes such as WeChat official account followers, app installers, product purchase customers, similar audience expansion, and relationship chain expansion, etc.

[0060] The content attribute subset may include, for example, ad title information, ad image information, ad video information, product database ID (Identity), product industry type, ad creative specifications, backend product ID, customer product ID, customer product name, product type, core product keywords, creative ID, custom product tags, and external product ID.

[0061] The above description, using an advertisement as an example, illustrates the contents of the attribute set associated with the target recommendation information, including the subset of attribution attributes, the subset of audience attributes, and the subset of content attributes. However, the embodiments disclosed herein are not limited to this. The attribute set associated with the target recommendation information may also include more other attributes, depending on the specific category of the recommendation information or on predetermined attribute filtering rules.

[0062] In step S210, based on the historical delivery data of the predetermined set of recommendation information containing the target recommendation information within the historical delivery period, multiple attributes are determined from the attribute set associated with the target recommendation information as multiple candidate aggregation dimensions, so that information for PCVR calibration of the target recommendation information can be obtained by aggregating the delivery data of the recommendation information under each candidate aggregation dimension.

[0063] According to an example of an embodiment of this disclosure, a dimensional optimization model can be established based on historical delivery data within a historical delivery period for a predetermined set of recommendation information containing target recommendation information. By solving the objective function of the dimensional optimization model, multiple optimal attributes can be determined from the attribute set of the target recommendation information as multiple candidate aggregation dimensions.

[0064] The predetermined recommendation information set is a collection of multiple recommendation information items to be calibrated (hereinafter referred to as predetermined recommendation information), and the target recommendation information can be any recommendation information in the predetermined recommendation information set. The predetermined recommendation information in the predetermined recommendation information set can be, for example, randomly determined; or they can belong to a common advertiser, that is, the predetermined recommendation information set can be a collection of all advertisements under a certain advertiser. In other words, the recommendation information processing method 200 according to the embodiments of this disclosure can perform PCVR calibration on all advertisements under a common advertiser; or they can have a common industry orientation, that is, the predetermined recommendation information set can be a collection of all advertisements under a certain industry orientation. In other words, the recommendation information processing method 200 according to the embodiments of this disclosure can perform PCVR calibration on all advertisements under a common industry orientation, etc. However, the embodiments of this disclosure are not limited to these, and the predetermined recommendation information set can also be a collection of recommendation information items to be calibrated selected according to any other rules.

[0065] Furthermore, since the delivery data aggregated under different aggregation dimensions has different impacts on the accuracy of PCVR calibration of the target recommendation information, this step can also determine the priority of each candidate aggregation dimension among multiple candidate aggregation dimensions. For example, the priority can be determined according to the magnitude of the impact of different aggregation dimensions on the accuracy of PCVR calibration of the target recommendation information. In the embodiments of this disclosure, a dimension optimization model can be constructed, and multiple candidate aggregation dimensions and the priority of each candidate aggregation dimension can be determined by solving the objective function of the dimension optimization model, as will be described in further detail below.

[0066] In step S220, a reference aggregation dimension for the target recommendation information is selected from the multiple candidate aggregation dimensions determined in step S210, based at least on the priority of each candidate aggregation dimension, to be used for PCVR calibration of the target recommendation information. For example, the candidate aggregation dimension with the highest priority can be selected as the reference aggregation dimension. However, in some cases, such as in the initial stage of recommendation information delivery, the delivery data of recommendation information under certain candidate aggregation dimensions may not be sufficient. In such cases, the data obtained by aggregating the delivery data of recommendation information under that candidate aggregation dimension has no effective reference value, i.e., the aggregated data is invalid. If the aggregated data of the candidate aggregation dimension with the highest priority among the multiple candidate aggregation dimensions is invalid, then using the aggregated data of the candidate aggregation dimension with the highest priority to calibrate the target recommendation information for PCVR may be inaccurate. Therefore, when selecting the reference aggregation dimension for the target recommendation information, in addition to priority, the validity of the aggregated data under each candidate aggregation dimension corresponding to the target recommendation information should also be considered.

[0067] According to an example of an embodiment of this disclosure, for each candidate aggregation dimension's set of recommended information, i.e., the set of all recommended information under each candidate aggregation dimension, the validity of the aggregated data under that candidate aggregation dimension can be determined based on the total delivery cost of the recommended information set. Delivery cost refers to the fee charged to the customer for delivering a certain amount or duration of recommended information. Since delivery cost can reflect the click-through rate or conversion rate of recommended information, it can indicate whether the recommended information has received sufficient exposure, and thus can be used to determine the validity of the aggregated data of the recommended information set. According to an example of an embodiment of this disclosure, if the total delivery cost of the recommended information set under a certain candidate aggregation dimension is greater than a first predetermined threshold, the aggregated data of that recommended information set can be determined to be valid; otherwise, the aggregated data of that recommended information set is determined to be invalid. The first predetermined threshold can be set according to actual circumstances, and this embodiment of the disclosure does not impose specific limitations on it.

[0068] Taking advertising as an example, ad spend refers to the fee charged to the advertiser for a certain amount of advertising or advertising over a certain period of time. A first predetermined threshold can be set based on the advertiser's target cost per conversion (target_cpa), where the advertiser's target cost per conversion is the cost the advertiser expects for each individual ad. For example, if the total ad spend in a certain ad set is greater than four times the target cost per conversion (4*target_cpa), the aggregated data for that ad set can be determined to be valid; otherwise, the aggregated data for that ad set is determined to be invalid.

[0069] Therefore, according to an example of an embodiment of this disclosure, selecting a reference aggregation dimension for target recommendation information from multiple candidate aggregation dimensions based at least on the priority of each candidate aggregation dimension may include: obtaining the total delivery cost of the set of recommendation information corresponding to the target recommendation information under each candidate aggregation dimension; and determining the candidate aggregation dimension with the highest priority and a total delivery cost of the set of recommendation information greater than a first predetermined threshold as the reference aggregation dimension. In other words, the candidate aggregation dimension with the highest priority and valid corresponding aggregation data among the multiple candidate aggregation dimensions is determined as the reference aggregation dimension to ensure that the most relevant and sufficiently sufficient aggregation data can be used to perform PCVR calibration on the target recommendation information.

[0070] After determining the reference aggregation dimension, in step S230, the estimated conversion rate and actual conversion rate of the reference recommendation information in the reference recommendation information set corresponding to the target recommendation information under that reference aggregation dimension are aggregated to obtain the total estimated conversion rate (PCVR_valid) and total actual conversion rate (Conv_valid) of the reference recommendation information set up to the current time in the current campaign period. Here, the set of all recommendation information under the reference aggregation dimension is called the reference recommendation information set. The campaign period can be any time period, such as 12h, 24h (i.e., a full day), a week, etc., which can be set according to actual needs, and this embodiment does not impose specific limitations on it. For example, for advertising, a day (24h) is usually used as a campaign period. The current time is the time when PCVR calibration of the target recommendation information is to be performed. For example, for an ad click at the current moment, after predicting the estimated conversion rate corresponding to this click using models such as PNN and FM, the total estimated conversion rate and total actual conversion rate generated by the reference recommendation information set before the current moment in the current campaign period can be obtained by aggregating the estimated conversion rate and actual conversion rate of all reference recommendation information in the reference recommendation information set. This can be used to calibrate the estimated conversion rate.

[0071] In step S240, a calibration parameter is determined based at least on the total estimated conversion rate and the total actual conversion rate of the reference recommendation information set, so that in step S250, the calibration parameter is used to calibrate the estimated conversion rate corresponding to the click on the target recommendation information at the current time.

[0072] Generally, in the initial stage of target recommendation information delivery, the delivery data for the target recommendation information itself is not yet sufficient. At this time, the quotient of the total actual conversion volume Conv_valid of the reference recommendation information set and the total estimated conversion rate PCVR_valid can be directly determined as the calibration parameter, that is, the calibration parameter f is:

[0073] f = Conv_valid / PCVR_valid (1)

[0074] As campaigns progress, the click-through rate and conversion rate of the target recommendation information itself continuously increase, generating substantial campaign data. At this point, the target recommendation information can be considered to have entered the mature stage of campaign execution. During this mature stage, the campaign data of the target recommendation information itself can be used for PCVR calibration. Furthermore, due to its better relevance, the target recommendation information's own data should play a more significant role in PCVR calibration compared to the data from the reference recommendation information set, thereby further improving the accuracy of PCVR calibration. Additionally, in some cases, such as for CPM (Cost Per Mille) ads, the predicted click-through rate (PCTR) also affects the estimated revenue of the ad. Therefore, when calibrating an ad's PCVR, considering the predicted PCTR simultaneously can effectively improve the accuracy of the estimated revenue determined using the calibrated PCVR.

[0075] Therefore, according to the example of the embodiment of this disclosure, before step S240, the recommendation information processing method 200 may further include determining whether the target recommendation information is in the initial stage or the mature stage of the current delivery cycle at the current moment based on predetermined rules. For example, the target recommendation information can be determined to be in the initial stage or the mature stage based on the current delivery cost and current conversion rate of the target recommendation information before the current moment in the current delivery cycle. Specifically, the total delivery cost and total conversion rate generated by the target recommendation information before the current moment in the current delivery cycle (which can be referred to as the current delivery cost and the current conversion rate, respectively) can be statistically determined. If the current delivery cost is less than or equal to a second predetermined threshold and the current conversion rate is less than or equal to a third predetermined threshold, the target recommendation information is determined to be in the initial stage of delivery; otherwise, the target recommendation information is determined to be in the mature stage of delivery. The second predetermined threshold and the third predetermined threshold can be determined according to the actual situation, and the embodiment of this disclosure does not impose specific restrictions on them. For example, taking advertising as an example, the second predetermined threshold can be 2 times the target conversion cost (2*target_cpa). As mentioned above, the advertiser's target conversion cost is the cost required for each conversion of a single advertisement that the advertiser expects; the third predetermined threshold can be, for example, 2. In other words, if the current campaign cost is less than or equal to 2 * target_cpa and the current conversion rate is less than or equal to 2, it can be determined that the target recommendation information is in the initial stage of campaigning; otherwise, it can be determined that the target recommendation information is in the mature stage of campaigning.

[0076] According to an example of an embodiment of this disclosure, when the target recommendation information is in the initial stage of delivery, the recommendation information processing method 200 may further include obtaining the total estimated clicks (PCTR_valid) and total actual clicks (ClickNum_valid) of the reference recommendation information set up to the current time within the current delivery period, and may at least determine calibration parameters based on the obtained total estimated conversions (PCVR_valid), total actual conversions (Conv_valid), total estimated clicks (PCTR_valid), and total actual clicks (ClickNum_valid) of the reference recommendation information set. In this case, the calibration parameter f can be determined as:

[0077] f=Conv_valid / (PCVR_valid*PCTR_valid / ClickNum_valid) (2)

[0078] When using the calibration parameters in the above formula to perform PCVR calibration on target recommendation information in the initial stage of delivery, in order to avoid the PCVR calibration process affecting the current delivery cost of the target recommendation information, an expansion coefficient (coef) determined based on the delivery cost of the target recommendation information can be further introduced. Then, the calibration parameters are determined based on this expansion coefficient (coef) and the total estimated conversion (PCVR_valid), total actual conversion (Conv_valid), total estimated clicks (PCTR_valid), and total actual clicks (ClickNum_valid) of the reference recommendation information set. At this time, the calibration parameter f can be further determined as:

[0079] f=coef*Conv_valid / (PCVR_valid* PCTR_valid / ClickNum_valid) (3)

[0080] According to an example of an embodiment of this disclosure, when the target recommendation information is in the mature stage of deployment, the recommendation information processing method 200 may further include: obtaining the estimated conversion volume (sum_pcvr), actual conversion volume (conversion_num), estimated click volume (sum_pctr), and actual click volume (clicknum) of the target recommendation information before the current time in the current deployment period. In this case, calibration parameters can be determined at least based on the total estimated conversion volume and total actual conversion volume of the reference recommendation information set, and the estimated conversion volume (sum_pcvr), actual conversion volume (conversion_num), estimated click volume (sum_pctr), and actual click volume (clicknum) of the target recommendation information before the current time in the current deployment period. As mentioned above, in PCVR calibration during the mature stage of deployment, the deployment data of the target recommendation information itself should play a more significant role, while the total estimated conversion volume and total actual conversion volume of the reference recommendation information set can be used as auxiliary factors. For example, the factor determined based on the total estimated conversion volume and total actual conversion volume of the reference recommendation information set can be called the historical PCVR calibration factor (history_pcvr_bias_factor). At this point, the calibration parameter f can be determined as:

[0081] f=(history_pcvr_bias_factor*conversion_num) / (sum_pcvr*sum_pctr / clicknum) (4)

[0082] Furthermore, in some cases, even during the mature stage of campaign deployment, the data for the target recommendation information may be relatively scarce. This could lead to the calibration parameters determined according to equation (4) being too high or too low, resulting in data anomalies. In such cases, a smoothing coefficient (smooth_base) can be introduced to smooth out these data anomalies. Specifically, the historical actual conversion volume of the target recommendation information within the historical campaign period can be obtained, and the smoothing coefficient can be determined based on the historical actual conversion volume of the target recommendation information and the length of time up to the current moment within the current campaign period. For example, the smoothing coefficient smooth_base can be determined using the following equation:

[0083]

[0084] Here, `sum_conversion_yesterday` represents the actual conversion rate of the target recommendation information in the previous campaign period, such as the total actual conversion rate yesterday; `H` represents the time length (in hours) from the start of the current campaign period to the current moment; the `ceil()` function rounds up; and the `min()` function takes the minimum value. In other words, the larger the historical actual conversion rate and the smaller the time length `H`, the larger the smoothing coefficient `smooth_base` will be.

[0085] After determining the smoothing coefficient, calibration parameters can be determined based on the smoothing coefficient, the total estimated and actual conversions of the reference recommendation information set, and the estimated conversions (sum_pcvr), actual conversions (conversion_num), estimated clicks (sum_pctr), and actual clicks (clicknum) of the target recommendation information up to the current time point within the current campaign period. For example, the calibration parameter f can be further determined as:

[0086] f=(history_pcvr_bias_factor*conversion_num+smooth_base) / (sum_pcvr*sum_pctr / clicknum+smooth_base) (6)

[0087] Among them, history_pcvr_bias_factor is a historical PCVR calibration factor determined based on the total estimated conversion volume and the total actual conversion volume of the reference recommendation information set. Its specific determination method will be described in detail below.

[0088] After determining the calibration parameters, in step S250, the estimated conversion rate corresponding to a click on the target recommendation information at the current time is calibrated using these calibration parameters. Specifically, at any given time, for the estimated conversion rate corresponding to a click at the current time obtained using a conversion rate prediction model such as PNN or FM, the estimated conversion rate can be calibrated by multiplying it by the calibration parameters. Alternatively, the calibration parameters can be used as a weighting factor in a conversion rate prediction model such as PNN or FM to directly generate the calibrated estimated conversion rate.

[0089] Below, in conjunction with Figure 3 Describe the process of determining multiple candidate aggregation dimensions and the priority of each candidate aggregation using the dimensionality optimization model in step S210 above. Figure 3A flowchart 300 illustrating an example of solving a dimensional optimization model according to an embodiment of this disclosure is shown. As mentioned earlier, in step S210, multiple candidate aggregation dimensions and the priority of each candidate aggregation dimension are determined based on a calibration analysis of the historical estimated conversion rate of a predetermined set of recommendation information containing target recommendation information within a historical delivery period. Assuming that the predetermined set of recommendation information contains n recommendation information items, including target recommendation information, a dimensional optimization model can be established, for example, as shown in equation (7):

[0090]

[0091]

[0092] Where i is the index of the recommended information in the recommended information set, i = 1...n; sumPcvr i convNum represents the sum of calibrated PCVR (which can be called calibrated estimated conversion volume) corresponding to valid clicks of the i-th recommendation within the historical delivery period. Valid clicks refer to clicks after removing erroneous clicks, fraudulent clicks, etc. i Let be the total conversion rate of the i-th recommendation within the historical campaign period; j is the index of each valid click for each recommendation within the historical campaign period, j = 1...m, where m is the total valid click rate of the recommendation within the historical campaign period; Pcvr j Let x1, x2, and x3 be the PCVR corresponding to the j-th valid click; x1, x2, and x3 are the candidate aggregation dimensions to be solved; f(x1, x2, x3) are the calibration parameters determined based on the candidate aggregation dimensions (x1, x2, x3).

[0093] Formulas (7)-(8) above define the dimensional optimization model. In other words, the purpose of the dimensional optimization model is to find the solution that makes the objective function... The smallest aggregation dimension (x1, x2, x3) is used as a candidate aggregation dimension, and the priority of the obtained candidate aggregation dimensions is solved simultaneously. It should be understood that although only three candidate aggregation dimensions (x1, x2, x3) are listed in formula (8), this is only an example. In the embodiments of this disclosure, more or fewer candidate aggregation dimensions can be set.

[0094] The following reference Figure 3 The specific steps for solving objective functions (7) and (8) will be described below.

[0095] In step S310, multiple attributes are randomly selected from the attribute set associated with the target recommendation information as multiple initial aggregation dimensions, each of which has a set priority. According to an example of an embodiment of this disclosure, at least one initial aggregation dimension can be selected from a subset of attribution attributes, a subset of audience attributes, and a subset of content attributes, respectively. For example, initial aggregation dimension x1 can be selected from the subset of attribution attributes, where x1 could be the advertiser; initial aggregation dimension x2 can be selected from the subset of audience attributes, where x2 could be the age of the target audience; and initial aggregation dimension x3 can be selected from the subset of content attributes, where x3 could be the type of advertised product. Furthermore, the priority order of x1, x2, and x3 can be set as x1>x2>x3, that is, x1 has the highest priority, x3 has the lowest priority, and x2 has a middle priority.

[0096] In step S320, multiple initial aggregation dimensions and corresponding set priorities are used to calibrate the historical estimated conversion rate of each historical click in the historical delivery period for each predetermined recommendation information in the predetermined recommendation information set.

[0097] Specifically, for each predetermined recommendation, an initial reference aggregation dimension is selected from multiple initial aggregation dimensions based on the set priority of each initial aggregation dimension. As mentioned above, when selecting the initial reference aggregation dimension for the predetermined recommendation, in addition to priority, the validity of the aggregated data under each initial candidate aggregation dimension should also be considered, and the validity of the aggregated data can be judged by using the total delivery cost of the recommendation set. According to an example of an embodiment of this disclosure, the total delivery cost of the recommendation set under multiple initial aggregation dimensions corresponding to the predetermined recommendation can be obtained first, and the initial candidate aggregation dimension with the highest priority and a total delivery cost greater than a first predetermined threshold can be determined as the initial reference aggregation dimension of the predetermined recommendation. The first predetermined threshold can be set according to the actual situation, for example, it can be 4*target_cpa, and this embodiment of the disclosure does not impose a specific limitation on it. For example, assuming that the aggregated data under initial aggregation dimensions x1, x2, and x3 are all valid for the predetermined recommendation, then initial aggregation dimension x1 is selected as the reference aggregation dimension of the predetermined recommendation.

[0098] Subsequently, for the predetermined recommendation information, the historical estimated conversion rate corresponding to each historical click of the predetermined recommendation information is calibrated based at least on the historical total estimated conversion and historical total actual conversion of the reference recommendation information set under the initial reference aggregation dimension corresponding to the predetermined recommendation information within the historical campaign period. For example, when the reference aggregation dimension of the predetermined recommendation information has been determined to be x1, the historical total estimated conversion and historical total actual conversion of the reference recommendation information set under x1 within the historical campaign period can be statistically determined. For example, when the campaign period is one day, the historical total estimated conversion and historical total actual conversion of the reference recommendation information set yesterday can be statistically determined. Then, for example, the calibration parameters can be calculated using (2)-(4) and (6) above to calibrate the historical estimated conversion rate corresponding to each historical click of the predetermined recommendation information.

[0099] Similarly, the historical estimated conversion rate for each pre-selected recommendation in the pre-selected recommendation information set is calibrated.

[0100] Next, in step S330, the deviation between the calibrated historical estimated conversion volume and the historical actual conversion volume of each predetermined recommendation information is calculated (this can be called the calibration deviation). The calibrated historical estimated conversion volume of each recommendation information is the sum of the calibrated historical estimated conversion rates corresponding to the effective historical clicks of that recommendation information within the historical delivery period. Furthermore, the cumulative sum of the calibration deviations of each recommendation information in the predetermined recommendation information set is calculated, i.e., the objective function in equation (7) above is calculated.

[0101] In step S340, the initial aggregation dimensions that minimize the objective function are determined as multiple candidate aggregation dimensions, and the priority of each of these initial aggregation dimensions is determined as the priority of the corresponding candidate aggregation dimension. In other words, after solving for (x1, x2, x3) as multiple candidate aggregation dimensions through the above steps, the priority of x1, x2, and x3 (e.g., x1>x2>x3) becomes the priority of the multiple candidate aggregation dimensions.

[0102] As can be seen in step S320 above, for each pre-defined recommendation information, its initial reference aggregation dimension is first determined, and then the historical delivery data under the determined initial reference aggregation dimension is statistically analyzed for use in the next calibration process. However, in practice, thanks to the rapid development of computer technology, parallel computing is often used to significantly improve the processing speed. Therefore, on the one hand, all delivery data under each initial aggregation dimension can be aggregated and statistically analyzed synchronously and in parallel; on the other hand, the total delivery cost of the recommendation information set under multiple initial aggregation dimensions corresponding to each pre-defined recommendation information can be determined synchronously and in parallel to determine the initial reference aggregation dimension of each pre-defined recommendation information. Thus, for any pre-defined recommendation information, once its initial reference aggregation dimension is determined, the aggregated data under its corresponding initial reference aggregation dimension can be quickly obtained for use in the next calibration process.

[0103] Therefore, when solving the objective function of the dimensionality optimization model, for the randomly selected initial aggregation dimensions x1, x2, and x3, the data of the historical delivery period can be processed according to the following example steps 1-7. In the following example steps 1-7, the recommended information is used as the advertisement, and the delivery period is 1 day (24h) as an example. That is, for any pre-determined advertisement to be calibrated in the pre-determined advertisement set, the multiple candidate aggregation dimensions and their corresponding priorities for PCVR calibration of the pre-determined advertisement in the new delivery period (e.g., today) can be determined by solving the objective function (7)-(8) of the dimensionality optimization model based on the historical delivery data of the pre-determined advertisement set in the historical delivery period (e.g., yesterday).

[0104] Specifically, for any historical calibration time within the historical delivery period, and for each pre-selected ad to be calibrated in the pre-selected ad set, the following steps can be used to calculate the historical calibration parameters used to calibrate the historical PCVR corresponding to the historical clicks of the pre-selected ad at that historical calibration time.

[0105] 1. Aggregate based on the initial aggregation dimension x1:

[0106] For each ad, calculate the total conversion volume (Conv_x1_hour) within the most recent hour from any historical calibration time within the historical campaign period, under different values ​​of x1. T The total effective clicks correspond to the sum of PCVR (PCVR_x1_hour). T Total clicks (ClickNum_x1_hour) T The sum of PCTR corresponding to the total effective exposure: PCTR_x1_hour TAnd the total conversions (Conv_x1_day), the sum of PCVR corresponding to the total effective clicks (PCVR_x1_day), the total clicks (ClickNum_x1_day), and the sum of PCTR corresponding to the total effective impressions (PCTR_x1_day) for the whole day.

[0107] Taking advertiser x1 as an example, this means calculating the delivery data (Conv_x1_hour) of all ads under different advertisers for the most recent hour at the historical calibration time. T PCVR_x1_hour T ClickNum_x1_hour T PCTR_x1_hour T , as well as the full-day delivery data Conv_x1_day, PCVR_x1_day, ClickNum_x1_day, and PCTR_x1_day.

[0108] Here, the total daily delivery data Conv_x1_day, PCVR_x1_day, ClickNum_x1_day, and PCTR_x1_day refer to the total delivery data generated at any historical calibration time within the historical delivery period. Since delivery data at different times before the historical calibration time has different impacts on the PCVR calibration at that historical calibration time, the following time decay strategy is adopted when calculating the total daily data:

[0109]

[0110] Where T represents any historical calibration time within the historical delivery cycle, such as any calibration time yesterday; t represents any time between the start of the historical delivery cycle and the historical calibration time. For example, if the historical delivery cycle is yesterday and delivery started at midnight yesterday, then t = 1...T; conv_x1_hour t This represents the total ad conversions between time t-1 and time t; PCVR_x1_hour t It is the sum of PCVR corresponding to the total effective ad clicks between time t-1 and time t; clicknum_x1_hour t This represents the total number of ad clicks between time t-1 and time t; PCTR_x1_hour t It is the sum of PCVR corresponding to the total effective ad exposures between time t-1 and time t; λ is the time decay coefficient, which can be 0.05 for example.

[0111] After obtaining the most recent hour's data and the entire day's data for different values ​​of x1 (e.g., different advertisers) at the historical calibration time, for any value of x11 (e.g., x1 represents the advertiser dimension, and x11 represents advertiser x11), the aggregated data under x11 can be determined from these two values. For example, the aggregated data under x11 can be determined according to the following rules:

[0112]

[0113] In other words, if the aggregated data for the most recent hour at the historical calibration time is sufficient, then the aggregated data for the most recent hour is selected as the aggregated data under x11; otherwise, the data for the entire day is selected as the aggregated data under x11. Here, the rule for judging whether the aggregated data for the most recent hour is sufficient is similar to the principle for judging whether the aggregated data is valid as described above. That is, if the total ad spending for the ad set under x11 is greater than the first predetermined threshold (e.g., 4*target_cpa) within the most recent hour at the historical calibration time, then the aggregated data for the most recent hour can be considered sufficient; otherwise, the aggregated data for the most recent hour is considered insufficient. The reason for performing the processing in the above formula (10) is that, theoretically, the closer the ad spending is to the calibration time, the better the correlation. Therefore, it is desirable to select aggregated data that is closer to the calibration time, such as the aggregated data for the most recent hour. However, if the aggregated data for the most recent hour is insufficient, it may lead to a decrease in the accuracy of PCVR calibration. Therefore, at this time, the more sufficient data for the entire day is selected as the final aggregated data.

[0114] 2. Aggregate based on the initial aggregation dimension x2 (this can be done simultaneously with step 1):

[0115] Similar to step 1, aggregated data Conv_x2, PCVR_x2, ClickNum_x2, and PCTR_x2 can be obtained for different values ​​of x2. For example, if x2 can be the age of the target audience, then aggregated data Conv_x2, PCVR_x2, ClickNum_x2, and PCTR_x2 for all ads at different age levels can be obtained. As mentioned earlier, step 2 can be processed in parallel with step 1 to improve processing speed.

[0116] 3. Aggregate based on the initial aggregation dimension x3 (this can be done simultaneously with steps 1 and 2):

[0117] Similar to steps 1 and 2, aggregated data Conv_x3, PCVR_x3, ClickNum_x3, and PCTR_x3 can be obtained for different values ​​of x3. For example, if x3 can be the type of advertising product, then aggregated data Conv_x3, PCVR_x3, ClickNum_x3, and PCTR_x3 for all advertisements under different advertising product types can be obtained. As mentioned earlier, step 3 can be processed synchronously and in parallel with steps 1 and 2 to improve processing speed.

[0118] 4. Determine the reference aggregation dimensions for pre-ordered advertisements.

[0119] Based on the x1, x2, and x3 dimension values ​​of the pre-ordered advertisement, for example, x11 (e.g., x11 indicates that the pre-ordered advertisement belongs to advertiser A), x21 (e.g., x21 indicates that the age of the target audience of the pre-ordered advertisement is 18), and x31 (e.g., x31 indicates that the product type of the pre-ordered advertisement is A), the corresponding aggregated data Conv_x11, PCVR_x11, ClickNum_x11, and PCTR_x11 are obtained from the aggregated data of different values ​​of x1 obtained in step 1. The corresponding aggregated data Conv_x21, PCVR_x21, ClickNum_x21, and PCTR_x21 are obtained from the aggregated data of different values ​​of x2 obtained in step 2. And the corresponding aggregated data Conv_x31, PCVR_x31, ClickNum_x31, and PCTR_x31 are obtained from the aggregated data of different values ​​of x3 obtained in step 3. Thus, the three types of aggregated data of the pre-ordered advertisement are obtained.

[0120] 5. Determine the initial reference aggregation dimension

[0121] In this step, the initial reference aggregation dimension can be determined based on the validity of the aggregated data under x11, x21 and x31 of the pre-defined advertisement, as well as the setting priority of the initial aggregation dimensions x1, x2 and x3.

[0122] As mentioned earlier, the initial aggregation dimensions x1, x2, and x3 are set with the priority x1>x2>x3. Therefore, we can first determine the validity of the aggregation data (Conv_x11, PCVR_x11, ClickNum_x11, PCTR_x11) corresponding to the x1 dimension value x11 of the pre-defined ad. The method for determining validity is similar to the principle mentioned above, that is, whether the total campaign cost corresponding to all ads under dimension value x11 is greater than a first predetermined threshold (e.g., 4*target_cpa). For example, whether the total campaign cost corresponding to all ads belonging to advertiser A is greater than the first predetermined threshold. If so, it means that the aggregation data corresponding to x11 is valid, and the initial aggregation dimension x1 can be directly determined as the initial reference aggregation dimension of the pre-defined ad; otherwise, it means that the aggregation data corresponding to x11 is invalid, and we continue to determine the validity of the aggregation data (Conv_x21, PCVR_x21, ClickNum_x21, PCTR_x21) corresponding to the x2 dimension value x21 of the pre-defined ad, and so on.

[0123] Ultimately, the highest-priority and valid aggregate data is retained and can be called the reference aggregate data (Conv_valid, PCVR_valid, ClickNum_valid, PCTR_valid), and its corresponding aggregate dimension is called the initial reference aggregate dimension. Furthermore, if all three types of aggregate data are invalid, the reference aggregate data can be used as the default value. For example, it can be the aggregate data corresponding to the highest-priority x1 dimension value x11 (Conv_x11, PCVR_x11, ClickNum_x11, PCTR_x11), meaning the default initial reference aggregate dimension is x1.

[0124] After determining the reference aggregated data, if the scheduled advertisement is in the initial stage of delivery at the historical calibration time, the historical calibration parameters used to calibrate the historical PCVR corresponding to the historical clicks of the scheduled advertisement at the historical calibration time can be determined through step 6.

[0125] 6. Calculate historical calibration parameters (if in the initial deployment phase).

[0126] If the scheduled advertisement is in the initial stage of delivery at the time of the historical calibration, the historical calibration parameter f can be calculated according to the following formula:

[0127] f=coef*Conv_valid / (PCVR_valid*PCTR_valid / ClickNum_valid) (11)

[0128] Here, coef is the scaling factor, used to balance the potential impact of PCVR calibration on ad spend. The scaling factor coef can be determined based on the planned ad spend. Specifically, ad spend can be divided into three phases: [0, 8*target_cpa); (8*target_cpa, 25*target_cpa]; [25*target_cpa, ∞). In these three phases, the scaling factor can be determined as follows:

[0129] ①[0, 8*target_cpa)

[0130] At this point, since the current advertising expenditure is relatively low and easily affected by interference, the expansion coefficient (coef) can be 1.

[0131] ②(8*target_cpa, 25*target_cpa]

[0132] At this stage, the expansion factor (coef) can be calculated using the following formula:

[0133]

[0134] ③[25*target_cpa,∞)

[0135] At this stage, the expansion factor (coef) can be calculated using the following formula:

[0136]

[0137] In equations (12)-(13) above, target_cpa is the advertiser's target conversion cost, and current_cpa_bias is the CPA deviation of the scheduled advertisement at the historical calibration time, which can be expressed as: current_cpa_bias=cost / (conv_num*target_cpa)–1, where cost is the consumption of the scheduled advertisement at the historical calibration time, and conv_num is the actual conversion amount of the scheduled advertisement at the historical calibration time.

[0138] Additionally, if the scheduled ad is in the mature stage of delivery at the historical calibration time, the historical calibration parameters used to calibrate the historical PCVR corresponding to the historical clicks of the scheduled ad at the historical calibration time can be determined through step 7.

[0139] 7. Calculate historical calibration parameters (if the system is in the mature deployment stage).

[0140] If the scheduled advertisement is in the mature stage of delivery at the time of the historical calibration, the historical calibration parameter f can be calculated according to the following formula:

[0141] f=(history_pcvr_bias_factor*conversion_num+smooth_base) / (sum_pcvr*sum_pctr / clicknum+smooth_base)

[0142]

[0143] Wherein, `history_pcvr_bias_factor` is the historical PCVR calibration factor; `sum_pcvr` is the sum of the estimated conversions corresponding to the total effective clicks of the scheduled ad before the historical calibration time within the historical campaign period; `conversion_num` is the actual conversion of the scheduled ad before the historical calibration time within the historical campaign period; `sum_pctr` is the sum of the estimated clicks corresponding to the total effective impressions of the scheduled ad before the historical calibration time within the historical campaign period; `clicknum` is the actual clicks of the scheduled ad before the historical calibration time within the historical campaign period; `smooth_base` is the smoothing coefficient used to smooth out data anomalies caused by possible insufficient data; `sum_conversion_yesterday` represents the actual conversion of the scheduled ad in the previous campaign period (i.e., the campaign period before this historical campaign). For example, if the historical campaign period is yesterday, then `sum_conversion_yesterday` represents the actual conversion of the scheduled ad the day before yesterday; `H` represents the time length (in hours) from the start of the historical campaign period to the historical calibration time; `ceil()` function represents rounding up; `min()` function represents taking the minimum value.

[0144] The historical PCVR calibration factor can be calculated based on the reference aggregate data (Conv_valid, PCVR_valid) determined in step 5. Further, the historical PCVR calibration factor can be calculated using the following formula:

[0145] history_pcvr_bias_factor=Conv_valid_p / PCVR_valid_p) / (Conv_valid / PCVR_valid) (15)

[0146] Here, Conv_valid_p and PCVR_valid_p are the reference aggregated data for the scheduled advertisements at spend tier p, respectively. Specifically, because the ad spend is relatively high during the mature campaign phase, and the spend varies significantly between different ads, the campaign data for ads with different spends will also have significant differences, thus having different impacts on the PCVR calibration of the scheduled advertisements.

[0147] Therefore, when performing aggregation statistics for the initial aggregation dimensions x1, x2, and x3, the ads under each aggregation dimension can be further divided into spending tiers, and the aggregation data of ads within different spending tiers under each initial aggregation dimension can be statistically analyzed. For example, the ad spending can be divided into 5 tiers as shown in column 2 of Table 1. For example, for the initial aggregation dimension x1, the total conversion volume of all ads whose spending falls into each spending tier under the initial aggregation dimension x1, as well as the sum of the PCVR corresponding to the total effective clicks, can be statistically analyzed. For example, the aggregation data obtained in tier 1 under the initial aggregation dimension x1 can be represented as (Conv_x1_1, PCVR_x1_1), and so on, as shown in Table 1.

[0148] Table 1 shows the aggregated data for each consumption tier under the initial reference aggregation dimension x1.

[0149] Class Cost of delivery The conversion rate and PCVR obtained from the statistics 1 Cost<=4*target_cpa Conv_x1_1, PCVR_x1_1 2 4*target_cpa<Cost<=10*target_cpa Conv_x1_2, PCVR_x1_2 3 10*target_cpa<Cost<=20*target_cpa Conv_x1_3, PCVR_x1_3 4 20*target_cpa<Cost<=40*target_cpa Conv_x1_4, PCVR_x1_4 5 Cost > 40 * target_cpa Conv_x1_5, PCVR_x1_5

[0150] Similarly, aggregated data for each consumption level can be obtained by targeting the initial reference aggregation dimension x2; and aggregated data for each consumption level can be obtained by targeting the initial reference aggregation dimension x3.

[0151] Next, based on the ad spend at the historical calibration time, the ad spend tier is determined. Assuming the ad spend tier p, and the ad's x1, x2, and x3 dimension values ​​are x11, x21, and x31 respectively, then the three types of aggregated data corresponding to the ad spend tier can be obtained from the aggregated data under the initial aggregated dimensions x1, x2, and x3: (Conv_x11_p, PCVR_x11_p), (Conv_x21_p, PCVR_x21_p), and (Conv_x31_p, PCVR_x31_p).

[0152] Then, similar to step 5 above, the initial reference aggregation dimension is determined based on the validity of the aggregated data under x11, x21, and x31 of the pre-determined advertisement, and the priority of the initial aggregation dimensions x1, x2, and x3. For example, assuming the determined initial reference aggregation dimension is x1, then the reference aggregation data (Conv_valid, PCVR_valid) corresponding to the pre-determined advertisement is (Conv_x11, PCVR_x11); and the corresponding aggregation data (Conv_valid_p, PCVR_valid_p) for each consumption level is (Conv_x11_p, PCVR_x11_p). At this time, the historical PCVR calibration factor can be calculated according to the above formula (15), and further, the historical calibration parameter f of the pre-determined advertisement in the mature stage of delivery can be calculated according to the above formula (14).

[0153] The above steps 1-7 describe the process of calculating the historical calibration parameters for PCVR calibration of any pre-selected ad in the pre-selected ad set at any historical calibration time within the historical campaign period. If the pre-selected ad is in the initial stage of campaigning at that historical calibration time, its historical calibration parameters can be calculated using step 6; if it is in the mature stage, its historical calibration parameters can be calculated using step 7. As mentioned earlier, the campaign spending and conversion volume of the pre-selected ad before the historical calibration time within the historical campaign period can be used to determine whether it is in the initial stage or the mature stage of campaigning. The specific determination rule is the same as the principle described in step S240 above, and will not be repeated here.

[0154] After determining the historical calibration parameters of the scheduled advertisement at the historical calibration time, these parameters can be used to calibrate the PCVR corresponding to the historical clicks of the scheduled advertisement at that historical calibration time. For example, the PCVR can be calibrated by multiplying the PCVR obtained by a conversion rate prediction model such as PNN or FM at the historical calibration time by the historical calibration parameters.

[0155] Through steps 1-7 above, the PCVR of historical clicks corresponding to the scheduled ad at the historical calibration time has been calibrated. Repeat the above process until the PCVR of all valid historical clicks for the scheduled ad within the historical campaign period has been calibrated, and the sum of the calibrated PCVRs corresponding to all valid historical clicks for the scheduled ad within the historical campaign period can be calculated (this can be called the historical estimated conversion rate), i.e., sumPcvr. i At this point, based on the historical actual conversion volume (convNum) of the scheduled advertisement within the historical campaign period. i This means that the deviation between the historical estimated conversion volume and the historical actual conversion volume can be calculated. For example, it can be expressed as (sumPcvr) i-convNum i ) 2 .

[0156] Repeat the above process until the PCVR calibration of all valid historical clicks for all pre-selected ads (i.e., ads 1...n) in the pre-selected ad set within the historical delivery period is completed, and the deviation of each pre-selected ad is calculated. This allows us to obtain the sum of the deviations of all pre-selected ads in the pre-selected ad set, which is the objective function.

[0157] Thus, one calculation cycle has been completed for the randomly selected initial aggregation dimensions (x1, x2, x3). The process continues by randomly selecting new initial aggregation dimensions (x1', x2', x3') and repeating the above calculation cycle. By exhausting all attributes in the attribute set, the initial aggregation dimensions that minimize the objective function are ultimately determined as candidate aggregation dimensions. Furthermore, the priority of each of these initial aggregation dimensions (e.g., x1>x2>x3) is determined as the priority of each corresponding candidate aggregation dimension. At this point, the objective function of the dimension optimization model has been solved.

[0158] For example, the particle swarm optimization (PSO) algorithm can be used to solve the objective function of the dimensionality optimization model. PSO is a collaborative random search algorithm that can quickly find the optimal solution in the solution space. However, this disclosure is not limited to this; any other suitable algorithm can be used to solve the objective function of the dimensionality optimization model according to this disclosure. For example, using PSO, three candidate aggregation dimensions x1, x2, and x3 can be obtained: the core product keyword, the sub-industry of the advertisement, and the category of interest, respectively, with the priority order being: core product keyword > sub-industry of the advertisement > category of interest. Then, in a new campaign period, these three candidate aggregation dimensions and their priorities can be used to calibrate the PCVR of the target recommendation information.

[0159] Additionally, it should be noted that when performing aggregation statistics on multiple initial aggregation dimensions (x1, x2, x3) in steps 1-3 above, further subdivisions such as "site set" and "new / old" can be made, and aggregation can be performed under these subdivided subdivisions (for example, similar to the aggregation based on consumption tiers mentioned above). Taking advertising as an example, "site set" refers to different advertising platforms, such as communication platforms and e-commerce platforms. Since the advertising data on different site sets have different macro characteristics, it will affect PCVR calibration. "New / old" refers to whether the advertisement is being advertised for the first time. For example, an advertisement that has been advertised since the current advertising cycle (e.g., today, or in the last two days) and has never been advertised before is a new advertisement; otherwise, it is an old advertisement. Since the advertising data of new and old advertisements also have different characteristics, it will also affect PCVR calibration.

[0160] Furthermore, it should be noted that the specific operations, parameters, formulas, etc. in steps 1-7 above are all examples and not intended to limit the embodiments of this disclosure. Various possible modifications based on steps 1-7 above are within the scope of the embodiments of this disclosure. For example, in step 7, a historical PCVR calibration factor is introduced by adding history_pcvr_bias_factor to formula (14), that is, historical data information of the associated advertisements of the predetermined advertisement is introduced. However, the embodiments of this disclosure are not limited to this, and the historical PCVR calibration factor can also be introduced in any other suitable manner such as weighting. As another example, in step 7, a click-through rate (CTR) calibration factor is introduced by adding sum_pctr / clicknum to formula (14). However, the embodiments of this disclosure are not limited to this, and the CTR calibration factor can also be introduced in any other suitable manner such as weighting, etc., and so on. These will not be described one by one here.

[0161] Therefore, aggregation can be further performed under sub-dimensions such as "site set" and "new / old". For example, assuming there are two different site sets A and B, in step 1 above, for the initial reference aggregation dimension x1, it can be further subdivided into four sub-dimensions: [A, new ad], [B, new ad], [A, old ad], and [B, old ad], and the aggregated data of all ads under these four sub-dimensions can be counted respectively. Similarly, in steps 2 and 3 above, for the initial aggregation dimensions x2 and x3, the aggregated data of all ads under these four sub-dimensions can also be counted respectively. Accordingly, when determining the reference aggregation dimension of the pre-selected ad in step 4 above, in addition to the x1, x2, and x3 dimension values ​​of the pre-selected ad, the aggregated data under the corresponding three sub-dimensions can also be selected based on which site set the pre-selected ad belongs to, and whether it is a new ad or an old ad. The aggregated data under these three sub-dimensions can then be used to calculate historical calibration parameters in subsequent steps. Since the process of calculating historical calibration parameters by subdividing sub-dimensions is similar to steps 1-7 above, it will not be repeated here for the sake of simplicity.

[0162] The above example, using advertisements as recommendation information, details the process of determining multiple candidate aggregation dimensions and their priorities for target recommendation information by solving the objective function of the dimensionality optimization model.

[0163] In the recommendation information processing method according to the embodiments of this disclosure, the specific implementation processes of determining the reference aggregation dimension in step S220, obtaining the delivery data of the reference recommendation information set corresponding to the target recommendation information in step S230, and determining the calibration parameters in step S240 are similar to those described in steps 1-7 above. The only difference is that: ① the initial aggregation dimensions x1, x2, and x3 in steps 1-3 are respectively set to the three candidate aggregation dimensions obtained by solving, such as the core product keyword, the sub-industry to which the advertisement belongs, and the industry category of interest, and the priority of the three candidate aggregation dimensions obtained by solving is adopted, such as the core product keyword > the sub-industry to which the advertisement belongs > the industry category of interest; ② the historical delivery period should be replaced with the current delivery period, and the historical calibration time should be replaced with the current calibration time. Therefore, the specific examples of steps S220 to S240 will not be repeated here.

[0164] In one example, the PCVR calibration of each advertisement in a predetermined advertisement set is performed using the recommendation information processing method 200 according to embodiments of the present disclosure, and the calibration results are shown in the appendix. Figure 4 To be continued Figure 7As shown in the table below. In the recommendation information processing method 200 used in this example, the objective function of the dimensionality optimization model is used to solve for three optimal candidate aggregation dimensions to calibrate the PCVR of the predetermined ad set. The statistical mean of the PCVR relative calibration deviation of the ads in the predetermined ad set is calculated by both the ad dimension and the advertiser dimension, as shown in the table below. Figure 4 and Figure 6 The third column (labeled MODE) shows the distribution of PCVR relative calibration deviations falling into each deviation interval, as shown in the figure. Figure 5 and Figure 7 The second row of each time period is shown (labeled MODE). The statistical mean of the PCVR relative calibration deviation can be, for example, the average value, mean square value, root mean square value, weighted value, etc. of the PCVR relative calibration deviation of each advertisement in the predetermined advertisement set. This embodiment of the disclosure does not impose specific limitations on this.

[0165] In addition, to highlight the advantages of Method 200, three additional candidate aggregation dimensions were artificially selected: advertiser, advertised product, and advertised product brand. The same PCVR calibration process was then performed on the same pre-defined ad set. The statistical mean of the PCVR relative to the calibration deviation of the ads in the calibrated pre-defined ad set is shown below. Figure 4 and Figure 6 The second column (labeled SET) shows the distribution ratio of PCVR relative calibration deviation falling into each deviation interval, for example... Figure 5 and Figure 7 The first row of each time period is shown (labeled SET).

[0166] from Figure 4-7 It can be seen that, compared with manually selecting candidate aggregation dimensions, the PCVR calibration is performed by using the recommendation information processing method 200 according to the embodiments of this disclosure to solve for the best candidate aggregation dimension using the objective function of the dimension optimization model. The statistical mean of the PCVR relative calibration deviation is effectively reduced, and the proportion of the PCVR relative calibration deviation falling into the overestimated deviation range (e.g., (1.5,∞)) or the underestimated deviation range (e.g., (0,0.5)) is also reduced. In other words, the proportion of PCVR relative calibration deviation that is too high or too low can be improved.

[0167] By utilizing the recommendation information processing method according to the above embodiments of this disclosure, a dimensional optimization model can be established based on historical delivery data within a historical delivery period for a predetermined set of recommendation information containing target recommendation information. By solving the objective function of the dimensional optimization model, multiple candidate aggregation dimensions that can achieve optimal PCVR calibration and the priority of each candidate aggregation dimension can be determined. This allows for precise PCVR calibration of the target recommendation information using multiple candidate aggregation dimensions and their priorities, greatly improving the accuracy of PCVR calibration. Furthermore, by utilizing the recommendation information processing method according to the embodiments of this disclosure, targeted calibration can be implemented by analyzing the characteristics of the estimated conversion rate of recommendation information for specific industries, specific audiences, specific content, etc., so as to more accurately deliver recommendation information, control delivery costs, predict delivery effects, etc.

[0168] Furthermore, the calibrated estimated conversion rate obtained using the recommendation information processing method according to the above embodiments of this disclosure can be used to sort the recommendation information to be delivered, so as to prioritize the delivery of recommendation information with higher estimated returns, thereby maximizing returns. See below for reference. Figure 8 A method for sorting recommendation information according to embodiments of this disclosure is described. Figure 8 A flowchart of a recommendation information sorting method 800 according to an embodiment of the present disclosure is shown.

[0169] like Figure 8 As shown, in step S810, the calibrated estimated conversion rate of each of the multiple recommended information to be delivered is obtained. The multiple recommended information to be delivered can be any type of recommended information to be delivered, such as multiple advertisements to be delivered; this embodiment of the disclosure does not impose specific limitations on this. In this embodiment of the disclosure, for example, the above reference can be used... Figure 2 The described recommendation information processing method calibrates the estimated conversion rate of each of the multiple recommendations to be delivered, thereby obtaining the calibrated estimated conversion rate of each of the multiple recommendations to be delivered. However, the embodiments of this disclosure are not limited to this, and the calibrated estimated conversion rate of each recommendation to be delivered can also be obtained in other ways. The estimated conversion rate of each recommendation to be delivered can be obtained, for example, through conversion rate prediction models such as PNN and FM as described above, or through any other method; the embodiments of this disclosure do not impose specific limitations on this. Since the above description has already detailed the use of... Figure 2 The recommended information processing method shown here calibrates the steps of the estimated conversion rate of the recommended information. Therefore, for the sake of simplicity, repeated descriptions of the same content are omitted here.

[0170] In step S820, the estimated revenue of each of the multiple recommended information to be delivered is calculated based on the calibrated estimated conversion rate of each of the multiple recommended information to be delivered. The following explanation uses advertising as an example. For instance, the estimated revenue of an advertising campaign can be measured using effective cost per mile (eCPM), but this embodiment is not limited to this; it can also be measured using any other metric such as return on investment (ROI). Typically, the eCPM of an advertisement can depend on the estimated conversion rate (PCVR), the estimated click-through rate (PCTR), and the advertiser's bid. The advertiser's bid refers to the fee the advertiser is willing to pay for one advertisement conversion. Therefore, the eCPM of an advertisement can be expressed as:

[0171] eCPM = PCVR × PCTR × bid

[0172] As can be seen from the above formula, the higher the estimated conversion rate (PCVR) of an ad, the higher its estimated revenue (eCPM), thus bringing higher revenue to the advertising platform. After obtaining the calibrated estimated conversion rate for each ad to be delivered using step S810, the above formula can be used to calculate a more accurate estimated revenue (eCPM) for each ad to be delivered. The estimated click-through rate (PCTR) can be obtained using known estimation methods in the art, while the advertiser's bid depends on the advertiser's real-time bid.

[0173] After obtaining the estimated revenue, such as eCPM, for each recommendation to be delivered, in step S830, the multiple recommendations to be delivered can be sorted based on the estimated revenue of each recommendation. For example, recommendations with higher estimated revenue can be ranked higher, while recommendations with lower estimated revenue can be ranked lower. After sorting, the multiple recommendations to be delivered can be delivered sequentially in a new delivery cycle according to the ranking results. The new delivery cycle can refer to a new delivery time period, a new delivery day, etc. For example, one or more recommendations ranked first can be selected for delivery; or, the top-ranked recommendations can be selected first, then the second-ranked recommendations can be selected, and so on.

[0174] By utilizing the recommendation information ranking method according to the above embodiments of this disclosure, the estimated revenue of the recommendation information to be delivered can be calculated more accurately based on the calibrated estimated conversion rate of the recommendation information to be delivered. Multiple recommendation information to be delivered can be ranked based on the estimated revenue, thereby prioritizing the delivery of recommendation information with higher estimated revenue, thereby maximizing revenue.

[0175] The following reference Figure 9 A recommendation information processing apparatus according to embodiments of the present disclosure is described. Figure 9 A schematic diagram of the structure of a recommendation information processing apparatus 900 according to an embodiment of the present disclosure is shown. Figure 9 As shown, the recommendation information processing device 900 may include a dimension determination unit 910, a selection unit 920, an aggregation unit 930, a calibration parameter determination unit 940, and a calibration unit 950. In addition to these five units, the calibration device 900 may also include other components; however, since these components are not relevant to the embodiments of this disclosure, their illustrations and descriptions are omitted here. Furthermore, since the function of the recommendation information processing device 900 is the same as described above... Figure 2 The details of the steps in the recommended information processing method 200 are similar, so for simplicity, repeated descriptions of some content are omitted here.

[0176] The dimension determination unit 910 is configured to establish a dimension optimization model based on historical delivery data within a historical delivery period for a predetermined set of recommendation information containing target recommendation information. By solving the objective function of the dimension optimization model, it determines multiple attributes as multiple candidate aggregation dimensions from the attribute set associated with the target recommendation information, and determines the priority of each candidate aggregation dimension. The target recommendation information can be any recommendation information to be calibrated for PCVR, such as advertisements, recommended news, article / video promotions, etc., and this embodiment does not impose specific limitations on it.

[0177] According to an example of an embodiment of this disclosure, the set of attributes associated with the target recommendation information may include at least an attribution attribute subset, an audience attribute subset, and a content attribute subset. As the names suggest, the attribution attribute subset includes multiple attributes related to the attribution of the recommendation information, the audience attribute subset includes multiple attributes related to the audience of the recommendation information, and the content attribute subset includes multiple attributes related to the content of the recommendation information.

[0178] The dimension determination unit 910 determines multiple attributes as multiple candidate aggregation dimensions from the attribute set associated with the target recommendation information based on the historical delivery data of the predetermined recommendation information set containing the target recommendation information within the historical delivery period. This allows information for PCVR calibration of the target recommendation information to be obtained by aggregating the delivery data of the recommendation information under each candidate aggregation dimension.

[0179] According to embodiments of this disclosure, a dimensional optimization model can be established based on historical delivery data within a historical delivery period for a predetermined set of recommendation information containing target recommendation information. By solving the objective function of the dimensional optimization model, multiple optimal attributes can be determined from the attribute set of the target recommendation information as multiple candidate aggregation dimensions.

[0180] The predetermined recommendation information set is a collection of multiple recommendation information items to be calibrated (hereinafter referred to as predetermined recommendation information), and the target recommendation information can be any recommendation information in the predetermined recommendation information set. The predetermined recommendation information in the predetermined recommendation information set can be, for example, randomly determined; or they can belong to a common advertiser, that is, the predetermined recommendation information set can be a collection of all advertisements under a certain advertiser. In other words, the recommendation information processing device 900 according to the embodiments of this disclosure can perform PCVR calibration on all advertisements under a common advertiser; or they can have a common industry orientation, that is, the predetermined recommendation information set can be a collection of all advertisements under a certain industry orientation. In other words, the recommendation information processing device 900 according to the embodiments of this disclosure can perform PCVR calibration on all advertisements under a common industry orientation, etc. However, the embodiments of this disclosure are not limited to these, and the predetermined recommendation information set can also be a collection of recommendation information items to be calibrated selected according to any other rules.

[0181] Furthermore, since the delivery data obtained by aggregating under different aggregation dimensions has different impacts on the accuracy of PCVR calibration of the target recommendation information, the dimension determination unit 910 can also determine the priority of each candidate aggregation dimension among multiple candidate aggregation dimensions. For example, the priority can be determined according to the magnitude of the impact of different aggregation dimensions on the accuracy of PCVR calibration of the target recommendation information. In this embodiment of the disclosure, a dimension optimization model can be constructed, and multiple candidate aggregation dimensions and the priority of each candidate aggregation dimension can be determined by solving the objective function of the dimension optimization model. The dimension optimization model can be defined, for example, by the above equations (7)-(8), and the process of solving the objective function of the dimension optimization model can be referred to the process described above in conjunction with steps 1-7, which will not be repeated here.

[0182] Selection unit 920 is configured to select a reference aggregation dimension for PCVR calibration of the target recommendation information from among the multiple candidate aggregation dimensions determined by dimension determination unit 910, based at least on the priority of each candidate aggregation dimension. For example, the candidate aggregation dimension with the highest priority can be selected as the reference aggregation dimension. However, in some cases, such as in the initial stage of recommendation information delivery, the delivery data for recommendation information under certain candidate aggregation dimensions may not be sufficient. In such cases, the data obtained by aggregating the delivery data for recommendation information under those candidate aggregation dimensions will not have effective reference value, i.e., the aggregated data is invalid. If the aggregated data of the candidate aggregation dimension with the highest priority among the multiple candidate aggregation dimensions is invalid, then using the aggregated data of the candidate aggregation dimension with the highest priority to calibrate the PCVR of the target recommendation information may be inaccurate. Therefore, when selecting the reference aggregation dimension for the target recommendation information, in addition to priority, the validity of the aggregated data under each candidate aggregation dimension corresponding to the target recommendation information should also be considered.

[0183] According to an example of an embodiment of this disclosure, for each candidate aggregation dimension's set of recommendation information, i.e., the set of all recommendation information under each candidate aggregation dimension, the selection unit 920 can also determine the validity of the aggregated data under that candidate aggregation dimension based on the total delivery cost of the recommendation information set. According to an example of an embodiment of this disclosure, if the total delivery cost of the recommendation information set under a certain candidate aggregation dimension is greater than a first predetermined threshold, it can be determined that the aggregated data of that recommendation information set is valid; otherwise, it is determined that the aggregated data of that recommendation information set is invalid. The first predetermined threshold can be set according to actual conditions, and this embodiment of the disclosure does not impose specific limitations on it.

[0184] Taking advertising as an example, ad spend refers to the fee charged to the advertiser for a certain amount of advertising or advertising over a certain period of time. A first predetermined threshold can be set based on the advertiser's target cost per conversion (target_cpa), where the advertiser's target cost per conversion is the cost the advertiser expects for each individual ad. For example, if the total ad spend in a certain ad set is greater than four times the target cost per conversion (4*target_cpa), the aggregated data for that ad set can be determined to be valid; otherwise, the aggregated data for that ad set is determined to be invalid.

[0185] Therefore, according to an example of an embodiment of this disclosure, the selection unit 920 is further configured to: obtain the total delivery cost of the set of recommendation information corresponding to the target recommendation information under each of the multiple candidate aggregation dimensions; and determine the candidate aggregation dimension with the highest priority and the total delivery cost of the set of recommendation information among the multiple candidate aggregation dimensions being greater than a first predetermined threshold as the reference aggregation dimension. That is, the candidate aggregation dimension with the highest priority and corresponding valid aggregation data among the multiple candidate aggregation dimensions is determined as the reference aggregation dimension to ensure that the most relevant and sufficiently sufficient aggregation data can be used to perform PCVR calibration on the target recommendation information.

[0186] After determining the reference aggregation dimension, the aggregation unit 930 aggregates the estimated and actual conversion rates of the reference recommendation information in the reference recommendation information set corresponding to the target recommendation information under that reference aggregation dimension, to obtain the total estimated conversion rate (PCVR_valid) and total actual conversion rate (Conv_valid) of the reference recommendation information set up to the current time in the current campaign period. Here, the set of all recommendation information under the reference aggregation dimension is called the reference recommendation information set. The campaign period can be any time period, such as 12 hours, 24 hours (i.e., a full day), a week, etc., which can be set according to actual needs, and this embodiment does not impose specific limitations on it. For example, for advertising, a day (24 hours) is usually used as a campaign period. The current time is the time when PCVR calibration of the target recommendation information is to be performed. For example, for an ad click at the current moment, after predicting the estimated conversion rate corresponding to this click using models such as PNN and FM, the aggregation unit 930 can aggregate the estimated conversion volume and actual conversion volume of all reference recommendation information in the reference recommendation information set to obtain the total estimated conversion volume and total actual conversion volume generated by the reference recommendation information set before the current moment in the current campaign period, so as to calibrate the estimated conversion rate.

[0187] The calibration parameter determination unit 940 is configured to determine calibration parameters based at least on the total estimated conversion rate and the total actual conversion rate of the reference recommendation information set, so that the calibration unit 950 can use the calibration parameters to calibrate the estimated conversion rate corresponding to the click on the target recommendation information at the current time.

[0188] Generally, in the initial stage of the delivery of target recommendation information, the delivery data of the target recommendation information itself is not yet sufficient. At this time, the quotient of the total actual conversion volume and the total estimated conversion rate of the reference recommendation information set can be directly determined as the calibration parameter. That is, the calibration parameter f can be determined by the above formula (1).

[0189] As campaigns progress, the click-through rate and conversion rate of the target recommendation information itself continuously increase, generating substantial campaign data. At this point, the target recommendation information can be considered to have entered the mature stage of campaign execution. During this mature stage, the campaign data of the target recommendation information itself can be used for PCVR calibration. Furthermore, due to its better relevance, the target recommendation information's own data should play a more significant role in PCVR calibration compared to the data from the reference recommendation information set, thereby further improving the accuracy of PCVR calibration. Additionally, in some cases, such as for CPM (Cost Per Mille) ads, the predicted click-through rate (PCTR) also impacts the estimated revenue of the ad. Therefore, when calibrating an ad's PCVR, considering the predicted PCTR simultaneously can effectively improve the accuracy of the estimated revenue determined using the calibrated PCVR.

[0190] Therefore, according to an example of an embodiment of this disclosure, the recommendation information processing device 900 may further include a delivery stage determination unit 960, which is configured to determine whether the target recommendation information is in the initial delivery stage or the mature delivery stage of the current delivery period at the current moment based on predetermined rules. For example, the delivery stage determination unit 960 may determine whether the target recommendation information is in the initial delivery stage or the mature delivery stage based on the current delivery consumption and current conversion volume of the target recommendation information before the current moment in the current delivery period. Specifically, the delivery stage determination unit 960 may statistically determine the total delivery consumption and total conversion volume generated by the target recommendation information before the current moment in the current delivery period (which may be referred to as the current delivery consumption and the current conversion volume, respectively), and determine that the target recommendation information is in the initial delivery stage when the current delivery consumption is less than or equal to a second predetermined threshold and the current conversion volume is less than or equal to a third predetermined threshold; otherwise, determine that the target recommendation information is in the mature delivery stage. The second predetermined threshold and the third predetermined threshold may be determined according to actual conditions, and this embodiment of the disclosure does not impose specific limitations on them. For example, taking advertising as an example, the second predetermined threshold can be twice the target conversion cost (2*target_cpa). As mentioned above, the advertiser's target conversion cost is the cost required for each conversion of a single ad that the advertiser expects. The third predetermined threshold can be, for example, 2. That is to say, when the current campaign cost is less than or equal to 2*target_cpa and the current conversion volume is less than or equal to 2, it can be determined that the target recommendation information is in the initial stage of campaigning; otherwise, it can be determined that the target recommendation information is in the mature stage of campaigning.

[0191] According to an example of an embodiment of this disclosure, when the target recommendation information is in the initial stage of delivery, the aggregation unit 930 can also obtain the total estimated clicks (PCTR_valid) and total actual clicks (ClickNum_valid) of the reference recommendation information set before the current time in the current delivery period, and the calibration parameter determination unit 940 can determine the calibration parameter based at least on the obtained total estimated conversions (PCVR_valid), total actual conversions (Conv_valid), total estimated clicks (PCTR_valid), and total actual clicks (ClickNum_valid) of the reference recommendation information set. At this time, the calibration parameter f can be determined by the above formula (2).

[0192] When using the calibration parameters in equation (2) above to perform PCVR calibration on target recommendation information in the initial stage of delivery, in order to avoid the PCVR calibration process affecting the current delivery cost of the target recommendation information, an expansion coefficient (coef) determined based on the delivery cost of the target recommendation information can be further introduced. Then, the calibration parameters are determined based on this expansion coefficient (coef) and the total estimated conversion (PCVR_valid), total actual conversion (Conv_valid), total estimated clicks (PCTR_valid), and total actual clicks (ClickNum_valid) of the reference recommendation information set. At this time, the calibration parameter f can be further determined by equation (3) above.

[0193] According to an example of an embodiment of this disclosure, when the target recommendation information is in the mature stage of deployment, the aggregation unit 930 can also obtain the estimated conversion volume (sum_pcvr), actual conversion volume (conversion_num), estimated click volume (sum_pctr), and actual click volume (clicknum) of the target recommendation information before the current time in the current deployment period. At this time, the calibration parameter determination unit 940 can determine the calibration parameters based at least on the total estimated conversion volume and total actual conversion volume of the reference recommendation information set, and the estimated conversion volume (sum_pcvr), actual conversion volume (conversion_num), estimated click volume (sum_pctr), and actual click volume (clicknum) of the target recommendation information before the current time in the current deployment period. As mentioned above, in PCVR calibration during the mature stage of deployment, the deployment data of the target recommendation information itself should play a more significant role, while the total estimated conversion volume and total actual conversion volume of the reference recommendation information set can be used as auxiliary factors. For example, the factor determined based on the total estimated conversion volume and total actual conversion volume of the reference recommendation information set can be called the historical PCVR calibration factor (history_pcvr_bias_factor). At this point, the calibration parameter f can be determined by the above equation (4).

[0194] Furthermore, in some cases, even during the mature stage of campaign deployment, the data for the target recommendation information may be relatively limited. This could lead to the calibration parameters determined according to equation (4) being too high or too low, resulting in data anomalies. In such cases, a smoothing coefficient (smooth_base) can be introduced to smooth out these data anomalies. Specifically, the historical actual conversion volume of the target recommendation information within the historical campaign period can be obtained, and the smoothing coefficient can be determined based on the historical actual conversion volume of the target recommendation information and the length of time up to the current moment within the current campaign period. For example, the smoothing coefficient smooth_base can be determined using equation (5) above.

[0195] After determining the smoothing coefficient, the calibration parameters can be determined based on the smoothing coefficient, the total estimated conversion rate and total actual conversion rate of the reference recommendation information set, and the estimated conversion rate (sum_pcvr), actual conversion rate (conversion_num), estimated click rate (sum_pctr), and actual click rate (clicknum) of the target recommendation information before the current time in the current campaign period. For example, the calibration parameter f can be further determined by the above equation (6).

[0196] After determining the calibration parameters, the calibration unit 950 can use these parameters to calibrate the estimated conversion rate corresponding to a click on the target recommendation information at the current time. Specifically, at any given time, after obtaining the estimated conversion rate corresponding to a click at the current time using a conversion rate prediction model such as PNN or FM, the estimated conversion rate can be calibrated by multiplying it by the calibration parameters. Alternatively, the calibration parameters can be used as a weighting factor in a conversion rate prediction model such as PNN or FM to directly generate the calibrated estimated conversion rate.

[0197] By utilizing the recommendation information processing apparatus according to the above embodiments of this disclosure, a dimensional optimization model can be established based on historical delivery data within a historical delivery period for a predetermined set of recommendation information containing target recommendation information. By solving the objective function of the dimensional optimization model, multiple candidate aggregation dimensions that can achieve optimal PCVR calibration and the priority of each candidate aggregation dimension can be determined. This allows for precise PCVR calibration of the target recommendation information using multiple candidate aggregation dimensions and their priorities, greatly improving the accuracy of PCVR calibration. Furthermore, by utilizing the recommendation information processing apparatus according to the embodiments of this disclosure, targeted calibration can be implemented by analyzing the characteristics of the estimated conversion rate of recommendation information for specific industries, specific audiences, specific content, etc., so as to more accurately deliver recommendation information, control delivery costs, predict delivery effects, etc.

[0198] The following reference Figure 10 A recommendation information sorting apparatus according to embodiments of the present disclosure is described. Figure 10 A schematic diagram of the structure of a recommendation information sorting apparatus 1000 according to an embodiment of the present disclosure is shown. Figure 10 As shown, the recommendation information sorting device 1000 includes an acquisition unit 1010, a revenue estimation unit 1020, and a sorting unit 1030. Besides these three units, the recommendation information sorting device 1000 may also include other components; however, since these components are not relevant to the content of this disclosure embodiment, their illustrations and descriptions are omitted here. Furthermore, since the function of the recommendation information sorting device 1000 is the same as described above... Figure 8 The details of the steps in the recommended information ranking method 800 are similar, so for simplicity, repeated descriptions of some content are omitted here.

[0199] The acquisition unit 1010 is configured to acquire the calibrated estimated conversion rate of each of a plurality of recommendation information to be delivered. The plurality of recommendation information to be delivered can be any set of recommendation information to be delivered, such as a plurality of advertisements to be delivered; this embodiment of the disclosure does not impose specific limitations on this. In this embodiment of the disclosure, for example, the acquisition unit 1010 can utilize the above reference... Figure 2 The described recommendation information processing method calibrates the estimated conversion rate of each of the multiple recommendations to be delivered, thereby obtaining the calibrated estimated conversion rate of each of the multiple recommendations to be delivered. However, the embodiments of this disclosure are not limited to this, and the calibrated estimated conversion rate of each recommendation to be delivered can also be obtained in other ways. The estimated conversion rate of each recommendation to be delivered can be obtained, for example, through conversion rate prediction models such as PNN and FM as described above, or through any other method; the embodiments of this disclosure do not impose specific limitations on this. Since the above description has already detailed the use of... Figure 2 The recommended information processing method shown here calibrates the steps of the estimated conversion rate of the recommended information. Therefore, for the sake of simplicity, repeated descriptions of the same content are omitted here.

[0200] The revenue estimation unit 1020 is configured to calculate the estimated revenue for each of the multiple recommended ads to be delivered, based on the calibrated estimated conversion rate of each of the multiple recommended ads to be delivered. The following explanation uses advertising as an example. For instance, the estimated revenue of an ad to be delivered can be measured using effective cost per mile (eCPM), but this embodiment is not limited to this and can also be measured using any other metric such as return on investment (ROI). Typically, the eCPM of an ad can depend on the estimated conversion rate (PCVR), the estimated click-through rate (PCTR), and the advertiser's bid. The advertiser's bid can refer to the cost the advertiser is willing to pay for one ad conversion. Therefore, the eCPM of an ad can be expressed as:

[0201] eCPM = PCVR × PCTR × bid

[0202] As can be seen from the above formula, the higher the estimated conversion rate (PCVR) of an ad, the higher its estimated revenue (eCPM), thus bringing higher revenue to the advertising platform. After the acquisition unit 1010 obtains the calibrated estimated conversion rate for each ad to be delivered, the above formula can be used to calculate a more accurate estimated revenue (eCPM) for each ad. The estimated click-through rate (PCTR) can be obtained using known estimation methods in the art, while the advertiser's bid depends on the advertiser's real-time bid.

[0203] After the revenue estimation unit 1020 obtains the estimated revenue, such as eCPM, for each recommendation to be delivered, the sorting unit 1030 can sort the multiple recommendations to be delivered based on the estimated revenue of each recommendation. For example, recommendations with higher estimated revenue can be ranked higher, while recommendations with lower estimated revenue can be ranked lower. After sorting, the sorting unit 1030 can deliver the multiple recommendations to be delivered sequentially in a new delivery cycle according to the sorting results. The new delivery cycle can refer to a new delivery time period, a new delivery day, etc. For example, one or more recommendations ranked first can be selected for delivery; or, the top-ranked recommendations can be selected first, then the second-ranked recommendations can be selected, and so on.

[0204] By utilizing the recommendation information sorting device according to the above embodiments of the present disclosure, the estimated revenue of the recommendation information to be delivered can be calculated more accurately based on the calibrated estimated conversion rate of the recommendation information to be delivered. Multiple recommendation information to be delivered can be sorted based on the estimated revenue, thereby prioritizing the delivery of recommendation information with higher estimated revenue, thereby maximizing revenue.

[0205] Furthermore, the device according to embodiments of this disclosure (e.g., a recommendation information processing device, a recommendation information sorting device, etc.) can also be used by means of Figure 11 The architecture of the exemplary computing device shown is used to implement this. Figure 11 A schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown. Figure 11 As shown, computing device 1100 may include a bus 1110, one or more CPUs 1120, read-only memory (ROM) 1130, random access memory (RAM) 1140, a communication port 1150 connected to a network, input / output components 1160, a hard disk 1170, etc. Storage devices in computing device 1100, such as ROM 1130 or hard disk 1170, may store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU. Computing device 1100 may also include a user interface 1180. Of course, Figure 11 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 11 One or more components in the computing device shown. The device according to embodiments of this disclosure can be configured to perform a recommendation information processing method and a recommendation information sorting method according to the various embodiments of this disclosure above, or to implement a recommendation information processing apparatus and a recommendation information sorting apparatus according to the various embodiments of this disclosure above.

[0206] The embodiments of this disclosure can also be implemented as a computer-readable storage medium. A computer-readable storage medium according to embodiments of this disclosure stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the recommendation information processing method and recommendation information sorting method according to embodiments of this disclosure, as described with reference to the above figures, can be performed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0207] According to embodiments of this disclosure, a computer program product or computer program is also provided, which includes computer-readable instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer-readable instructions from the computer-readable storage medium and execute the computer-readable instructions, causing the computer device to perform the recommendation information processing method and recommendation information ranking method described in the various embodiments above.

[0208] Those skilled in the art will understand that the contents disclosed herein can be varied and modified in many ways. For example, the various devices or components described above can be implemented in hardware, or in software, firmware, or a combination of some or all of the three.

[0209] Furthermore, as shown in this disclosure and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "comprising" or "including" and similar terms mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, without excluding other elements or objects. The terms "connected" or "linked" and similar terms are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0210] Furthermore, flowcharts are used in this disclosure to illustrate the operations performed by the system according to embodiments of this disclosure. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously. Additionally, other operations can be superimposed on these processes, or one or more steps can be removed from these processes.

[0211] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It should also be understood that terms such as those defined in a common dictionary shall be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0212] The present disclosure has been described in detail above; however, it will be apparent to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered ways without departing from the spirit and scope defined by the claims. Therefore, the description herein is for illustrative purposes only and is not intended to be restrictive.

Claims

1. A method for processing recommendation information, comprising: Based on the dimensional optimization model established for the historical delivery data of the predetermined set of recommendation information containing target recommendation information within the historical delivery period, by solving the objective function of the dimensional optimization model, multiple attributes are determined from the attribute set associated with the target recommendation information as multiple candidate aggregation dimensions, and the priority of each candidate aggregation dimension among the multiple candidate aggregation dimensions is determined. Based at least on the priority of each of the plurality of candidate aggregation dimensions, a reference aggregation dimension for the target recommendation information is selected from the plurality of candidate aggregation dimensions; By aggregating the estimated conversion volume and actual conversion volume of the reference recommendation information in the reference recommendation information set corresponding to the target recommendation information under the reference aggregation dimension, the total estimated conversion volume and total actual conversion volume of the reference recommendation information set before the current moment in the current campaign period are obtained. The calibration parameters are determined based at least on the total estimated conversion rate and the total actual conversion rate of the reference recommendation information set; as well as The calibration parameters are used to calibrate the estimated conversion rate corresponding to the click on the target recommendation information at the current time. The objective function of the dimensional optimization model is the sum of the calibration deviations of the historical estimated conversion rates of the predetermined recommendation information in the predetermined recommendation information set within the historical delivery period. Specifically, by solving the objective function of the dimensional optimization model, multiple attributes are determined from the attribute set associated with the target recommendation information as multiple candidate aggregation dimensions, and the priority of each candidate aggregation dimension is determined, including: Multiple attributes are randomly selected from the set of attributes associated with the target recommendation information as multiple initial aggregation dimensions, and each of the multiple initial aggregation dimensions has a set priority; Using the multiple initial aggregation dimensions and the set priority, the historical estimated conversion rate corresponding to each historical click of each predetermined recommendation information in the predetermined recommendation information set within the historical delivery period is calibrated. Calculate the calibration deviation between the calibrated historical estimated conversion volume and the historical actual conversion volume for each pre-defined recommendation information, and sum the calibration deviations of the estimated conversion volumes of the pre-defined recommendation information in the set over the historical campaign period, using this sum as the objective function; and The multiple initial aggregation dimensions that minimize the objective function are determined as the multiple candidate aggregation dimensions, and the priority of each initial aggregation dimension among the multiple initial aggregation dimensions that minimize the objective function is determined as the priority of each corresponding candidate aggregation dimension.

2. The recommendation information processing method according to claim 1, wherein, Using the multiple initial aggregation dimensions and the set priority, calibrating the historical estimated conversion rate corresponding to each historical click of each predetermined recommendation in the predetermined recommendation information set includes: For each piece of pre-recommendation information in the pre-recommendation information set: An initial reference aggregation dimension is selected from the plurality of initial aggregation dimensions based at least on the set priority of each initial aggregation dimension; Historical calibration parameters are determined at least based on the historical total estimated conversion volume and historical total actual conversion volume of the reference recommendation information set under the initial reference aggregation dimension corresponding to the predetermined recommendation information within the historical campaign period; and Using the historical calibration parameters, the historical estimated conversion rate corresponding to each historical click of the predetermined recommendation information within the historical delivery period is calibrated.

3. The recommendation information processing method according to claim 1, wherein, Based at least on the priority of each of the plurality of candidate aggregation dimensions, the reference aggregation dimension for selecting the target recommendation information from the plurality of candidate aggregation dimensions includes: Obtain the total delivery cost of the recommendation information set corresponding to the target recommendation information under each of the multiple candidate aggregation dimensions; and The candidate aggregation dimension with the highest priority among the multiple candidate aggregation dimensions whose total delivery cost of the recommended information set is greater than a first predetermined threshold is determined as the reference aggregation dimension of the target recommended information.

4. The recommendation information processing method according to claim 1, wherein, Before determining calibration parameters based at least on the total estimated conversions and the total actual conversions from the reference recommendation information set, the recommendation information processing method further includes: Based on predetermined rules, it is determined whether the target recommendation information is currently in the initial or mature stage of the current campaign period. Wherein, when the target recommendation information is in the initial stage of delivery, the recommendation information processing method further includes: obtaining the total estimated clicks and total actual clicks of the reference recommendation information set before the current time within the current delivery period, and Wherein, when the target recommendation information is in the mature stage of deployment, the recommendation information processing method further includes: obtaining the estimated conversion volume, actual conversion volume, estimated click volume and actual click volume of the target recommendation information before the current time in the current deployment period.

5. The recommendation information processing method according to claim 4, wherein, Determining whether the target recommendation information is in the initial or mature stage of the current campaign based on predetermined rules includes: Determine the current campaign spending and current conversion rate generated by the target recommendation information before the current time point within the current campaign period; When the current campaign expenditure is less than or equal to a second predetermined threshold and the current conversion rate is less than or equal to a third predetermined threshold, the target recommendation information is determined to be in the initial stage of the campaign; otherwise, the target recommendation information is determined to be in the mature stage of the campaign.

6. The recommendation information processing method according to claim 4, wherein, When the target recommendation information is in the initial stage of deployment, determining calibration parameters based at least on the total estimated conversion rate and the total actual conversion rate of the reference recommendation information set includes: The calibration parameters are determined based at least on the total estimated conversions, total actual conversions, total estimated clicks, and total actual clicks of the reference recommendation information set.

7. The recommendation information processing method according to claim 6, wherein, The calibration parameters are determined based at least on the total estimated conversions, total actual conversions, total estimated clicks, and total actual clicks of the reference recommendation information set, including: Obtain the target recommendation information and the current campaign spending generated before the current time in the current campaign period, and determine the expansion coefficient based on the current campaign spending; and The calibration parameters are determined based on the amplification factor and the total estimated conversions, total actual conversions, total estimated clicks, and total actual clicks of the reference recommendation information set.

8. The recommendation information processing method according to claim 4, wherein, When the target recommendation information is in the mature stage of deployment, determining the calibration parameters based at least on the total estimated conversion rate and the total actual conversion rate of the reference recommendation information set includes: The calibration parameters are determined based at least on the total estimated conversion and total actual conversion of the reference recommendation information set, and the estimated conversion, actual conversion, estimated clicks, and actual clicks of the target recommendation information before the current time in the current campaign period.

9. The recommendation information processing method according to claim 8, wherein, The calibration parameters are determined based at least on the total estimated conversions and total actual conversions of the reference recommendation information set, and the estimated conversions, actual conversions, estimated clicks, and actual clicks of the target recommendation information before the current time in the current campaign period, including: Obtain the historical actual conversion volume of the target recommendation information within the historical campaign period, and determine the smoothing coefficient based on the historical actual conversion volume of the target recommendation information and the length of time up to the current moment within the current campaign period; The calibration parameters are determined based on the smoothing coefficient, the total estimated conversion rate and total actual conversion rate of the reference recommendation information set, and the estimated conversion rate, actual conversion rate, estimated click rate, and actual click rate of the target recommendation information before the current time in the current campaign period.

10. The recommendation information processing method according to claim 1, wherein, The attribute set includes at least a subset of attribution attributes, a subset of audience attributes, and a subset of content attributes of the target recommendation information, and determines multiple attributes as multiple candidate aggregation dimensions from the attribute set associated with the target recommendation information, including: At least one attribute is determined from the subset of attribution attributes, the subset of audience attributes, and the subset of content attributes as a candidate aggregation dimension among the plurality of candidate aggregation dimensions.

11. A method for ranking recommendation information, comprising: Obtain the calibrated estimated conversion rate for each of the multiple recommended messages to be delivered; Based on the calibrated estimated conversion rate of each of the plurality of recommended information to be delivered, calculate the estimated revenue of each of the plurality of recommended information to be delivered; as well as Based on the estimated revenue of each of the plurality of recommendations to be delivered, the plurality of recommendations to be delivered are sorted, and the plurality of recommendations to be delivered are delivered sequentially according to the sorting result. The step of obtaining the calibrated estimated conversion rate for each of the multiple recommendations to be delivered includes: The estimated conversion rate of each of the plurality of recommended information to be delivered is calibrated using the method described in any one of claims 1-10, so as to obtain the calibrated estimated conversion rate of each of the plurality of recommended information to be delivered.

12. A recommendation information processing apparatus, comprising: The dimension determination unit is configured to establish a dimension optimization model based on historical delivery data within a historical delivery period for a predetermined set of recommendation information containing target recommendation information. By solving the objective function of the dimension optimization model, it determines multiple attributes as multiple candidate aggregation dimensions from the set of attributes associated with the target recommendation information, and determines the priority of each candidate aggregation dimension among the multiple candidate aggregation dimensions. The selection unit is configured to select a reference aggregation dimension from the plurality of candidate aggregation dimensions based at least on the priority of each of the plurality of candidate aggregation dimensions; The aggregation unit is configured to aggregate the estimated conversion volume and actual conversion volume of the reference recommendation information in the reference recommendation information set corresponding to the target recommendation information under the reference aggregation dimension, and obtain the total estimated conversion volume and total actual conversion volume of the reference recommendation information set before the current time in the current campaign period. The calibration parameter determination unit is configured to determine calibration parameters based at least on the total estimated conversion and the total actual conversion of the reference recommendation information set. as well as The calibration unit is configured to calibrate the estimated conversion rate corresponding to a click on the target recommendation information at the current time using the calibration parameters. The objective function of the dimensional optimization model is the sum of the calibration deviations of the historical estimated conversion rates of the predetermined recommendation information in the predetermined recommendation information set within the historical delivery period. The dimension determination unit is further configured as follows: Multiple attributes are randomly selected from the set of attributes associated with the target recommendation information as multiple initial aggregation dimensions, and each of the multiple initial aggregation dimensions has a set priority; Using the multiple initial aggregation dimensions and the set priority, the historical estimated conversion rate corresponding to each historical click of each predetermined recommendation information in the predetermined recommendation information set within the historical delivery period is calibrated. Calculate the calibration deviation between the calibrated historical estimated conversion volume and the historical actual conversion volume for each pre-defined recommendation information, and sum the calibration deviations of the estimated conversion volumes of the pre-defined recommendation information in the set over the historical campaign period, using this sum as the objective function; and The multiple initial aggregation dimensions that minimize the objective function are determined as the multiple candidate aggregation dimensions, and the priority of each initial aggregation dimension among the multiple initial aggregation dimensions that minimize the objective function is determined as the priority of each corresponding candidate aggregation dimension.

13. A recommendation information processing device, comprising: One or more processors; as well as One or more memories, wherein computer-readable code is stored in the memories, and when executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described in any one of claims 1-10.

Images