Information Display Method, Device, Computer Equipment and Storage Medium

Abstract

This application relates to an information display method, apparatus, computer device, and storage medium, and pertains to the field of Internet application technologies. The method includes: obtaining the actual display results of each displayable information; obtaining the predicted display results of each displayable information corresponding to at least two sub-ranking models; obtaining the calibration coefficients of each displayable information corresponding to at least two sub-ranking models based on the actual display results of each displayable information and the predicted display results of each displayable information corresponding to at least two sub-ranking models; in subsequent time, displaying each displayable information according to the displayable information ranking model and the calibration coefficients of each displayable information corresponding to at least two sub-ranking models. By displaying each displayable information based on the ranking results of the calibrated displayable information ranking model, the accuracy of the displayable information ranking model is improved, and thus the display effect of each displayable information is enhanced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet application technologies, and in particular, to an information display method, apparatus, computer device, and storage medium. Background Art

[0002] With the continuous development of network and computer technologies, online advertising has become an effective means for media owners to convert user traffic into cash revenue. In order to improve the accuracy of online advertising placement, after sorting various online advertisements through a sorting model, the display can be selected according to the sorting results.

[0003] In the related art, during the training process of the sorting model, a single model that composes the sortable information sorting model is usually calibrated by using a validation data set for model evaluation. For example, the model training device can obtain a training data set and a validation data set, train a sorting model through the training data set, and then input the validation data set into the sorting model to calibrate the sorting model.

[0004] However, the solution shown in the related art is only applicable to calibrating a single-level sorting model through a pre-acquired validation data set during the training process, is not applicable to a multi-level sorting model, and the model cannot be changed during the subsequent model application process, resulting in poor accuracy of the sorting model. Summary of the Invention

[0005] The embodiments of the present application provide an information display method, apparatus, computer device, and storage medium, which can improve the accuracy of sorting and displaying sortable information. The technical solution is as follows:

[0006] On the one hand, an information display method is provided. The method includes:

[0007] Obtain the actual display results of each piece of sortable information; the sortable information is the information selected for display by predicting the probability of receiving a specified user operation through a sortable information sorting model, and the sortable information sorting model includes at least two cascaded sub-sorting models, and the actual display results refer to the actual data received after each piece of sortable information is displayed on the display platform and receives the specified user operation;

[0008] Obtain the predicted display results of each piece of sortable information corresponding to the at least two sub-sorting models, where the predicted display results are used to indicate the predicted probabilities of the at least two sub-sorting models for predicting that each piece of sortable information receives a specified user operation;

[0009] Obtain the calibration coefficients corresponding to each of the at least two sub-ranking models for each displayable information based on the actual display results of each displayable information and the predicted display results of each displayable information corresponding to the at least two sub-ranking models respectively;

[0010] Based on the calibration coefficients of the at least two sub-ranking models, correct the predicted probabilities of the at least two sub-ranking models for each displayable information subsequently.

[0011] On the other hand, an information display device is provided, and the device includes:

[0012] A first acquisition module for acquiring the actual display results of each displayable information; the displayable information is information selected for display by predicting the probability of receiving a specified user operation through a displayable information ranking model and according to the predicted probability; the displayable information ranking model includes at least two cascaded sub-ranking models, and the actual display result refers to the actual data of each displayable information received after being displayed on the display platform and receiving the specified user operation;

[0013] A second acquisition module for acquiring the predicted display results of each displayable information corresponding to the at least two sub-ranking models respectively, where the predicted display results are used to indicate the predicted probabilities of the at least two sub-ranking models for each displayable information to receive a specified user operation;

[0014] A third acquisition module for obtaining the calibration coefficients corresponding to each of the at least two sub-ranking models for each displayable information based on the actual display results of each displayable information and the predicted display results of each displayable information corresponding to the at least two sub-ranking models respectively;

[0015] A display module for correcting the predicted probabilities of the at least two sub-ranking models for each displayable information subsequently based on the calibration coefficients of the at least two sub-ranking models.

[0016] Optionally, the at least two sub-ranking models include a rough selection ranking model and a refined selection ranking model cascaded after the rough selection ranking model;

[0017] The second acquisition module includes:

[0018] A first acquisition sub-module for obtaining the calibration coefficients corresponding to each of the refined selection ranking models for each displayable information based on the actual display results of each displayable information and the predicted display results of each displayable information corresponding to the refined selection ranking model respectively;

[0019] A second acquisition sub-module, configured to obtain, based on the predicted display results of the respective displayable information corresponding to the refined sorting model and the predicted display results of the respective displayable information corresponding to the rough sorting model, the calibration coefficients of the respective displayable information corresponding to the rough sorting model.

[0020] Optionally, the first acquisition sub-module includes:

[0021] A first acquisition unit, configured to obtain, based on the actual display results of the respective displayable information and the predicted display results of the respective displayable information corresponding to the refined sorting model, an initial value of the first calibration coefficient of the refined sorting model and the target values of the calibration coefficients of the respective displayable information corresponding to the refined sorting model;

[0022] A second acquisition unit, configured to obtain, based on the initial value of the first calibration coefficient and the target values of the calibration coefficients of the respective displayable information corresponding to the refined sorting model, the calibration coefficients of the respective displayable information corresponding to the refined sorting model.

[0023] Optionally, the first acquisition unit includes:

[0024] A first acquisition subunit, configured to obtain, based on the actual display results of the respective displayable information in a first time period and the predicted display results of the respective displayable information corresponding to the refined sorting model in the first time period, an initial value of the first calibration coefficient of the refined sorting model;

[0025] A second acquisition subunit, configured to obtain, based on the actual display results of the respective displayable information in a second time period, the target values of the calibration coefficients of the respective displayable information corresponding to the refined sorting model.

[0026] Optionally, the actual display results include actual exposure counts, actual click counts, and actual conversion counts; the predicted display results include estimated click-through rates and estimated conversion rates; the calibration coefficients include click-through rate calibration coefficients and conversion rate calibration coefficients;

[0027] The first acquisition subunit is configured to:

[0028] Obtain the sum of the actual click counts of the respective displayable information in the first time period and the sum of the actual conversion counts of the respective displayable information in the first time period;

[0029] Based on the actual exposure counts of the respective displayable information in the first time period, and the estimated click-through rate and estimated conversion rate of the respective displayable information corresponding to the selected sorting model in the first time period, obtain the sum of the first estimated click counts of the respective displayable information corresponding to the selected sorting model in the first time period, and the sum of the first estimated conversion counts of the respective displayable information in the first time period;

[0030] Based on the sum of the actual click counts of the respective displayable information in the first time period and the sum of the first estimated click counts, obtain the initial value of the first click-through rate calibration coefficient of the selected sorting model;

[0031] Based on the sum of the actual conversion counts of the respective displayable information in the first time period and the sum of the first estimated conversion counts, obtain the initial value of the first conversion rate calibration coefficient of the selected sorting model.

[0032] Optionally, the actual display results include actual exposure counts, actual click counts, and actual conversion counts; the predicted display results include estimated click-through rates and estimated conversion rates; the calibration coefficients include click-through rate calibration coefficients and conversion rate calibration coefficients;

[0033] The second acquisition subunit is used for:

[0034] Obtain the actual click counts of the respective displayable information in the second time period, and the actual conversion counts of the respective displayable information in the second time period;

[0035] Based on the actual exposure counts of the respective displayable information in the second time period, and the estimated click-through rate and estimated conversion rate of the respective displayable information corresponding to the selected sorting model in the second time period, obtain the estimated click counts and estimated conversion counts of the respective displayable information corresponding to the selected sorting model in the second time period;

[0036] Based on the actual click counts of the respective displayable information in the second time period and the estimated click counts of the respective displayable information corresponding to the selected sorting model in the second time period, obtain the target values of the click-through rate calibration coefficients of the respective displayable information corresponding to the selected sorting model;

[0037] Based on the actual conversion counts of the respective displayable information in the second time period and the estimated conversion counts of the respective displayable information corresponding to the selected sorting model in the second time period, obtain the target values of the conversion rate calibration coefficients of the respective displayable information corresponding to the selected sorting model.

[0038] Optionally, the second obtaining unit is configured to perform weighted summation on the initial value of the first calibration coefficient and the target values of the calibration coefficients of the respective displayable information corresponding to the refined sorting model, so as to obtain the calibration coefficients of the respective displayable information corresponding to the refined sorting model.

[0039] Optionally, the second obtaining sub-module includes:

[0040] The third obtaining unit is configured to obtain the initial value of the second calibration coefficient of the rough sorting model and the target values of the calibration coefficients of the respective displayable information corresponding to the initial sorting model based on the predicted display results of the respective displayable information corresponding to the refined sorting model and the predicted display results of the respective displayable information corresponding to the rough sorting model;

[0041] The fourth obtaining unit is configured to obtain the calibration coefficients of the respective displayable information corresponding to the rough sorting model based on the initial value of the second calibration coefficient and the target values of the calibration coefficients of the respective displayable information corresponding to the initial sorting model.

[0042] Optionally, the third obtaining unit includes:

[0043] The third obtaining sub-unit is configured to obtain the initial value of the second calibration coefficient of the rough sorting model based on the predicted display results of the respective displayable information corresponding to the refined sorting model in the third time period and the predicted display results of the respective displayable information corresponding to the rough sorting model in the third time period;

[0044] The fourth obtaining sub-unit is configured to obtain the target values of the calibration coefficients of the respective displayable information corresponding to the rough sorting model based on the predicted display results of the respective displayable information corresponding to the refined sorting model in the fourth time period.

[0045] Optionally, the predicted display result includes an estimated click-through rate and an estimated conversion rate; the calibration coefficient includes a click-through rate calibration coefficient and a conversion rate calibration coefficient;

[0046] The third obtaining sub-unit is configured to obtain the sum of the second estimated click-through numbers of the respective displayable information corresponding to the refined sorting model in the third time period and the sum of the second estimated conversion numbers of the respective displayable information corresponding to the refined sorting model in the third time period based on the actual exposure numbers of the respective displayable information in the third time period, and the estimated click-through rate and the estimated conversion rate of the respective displayable information corresponding to the refined sorting model in the third time period;

[0047] Based on the actual exposure counts of the respective displayable information in the third time period, and the estimated click-through rate and estimated conversion rate of the respective displayable information corresponding to the rough sorting model in the third time period, obtain the sum of the third estimated click counts of the respective displayable information corresponding to the rough sorting model in the third time period, and the sum of the third estimated conversion counts of the respective displayable information in the third time period;

[0048] Based on the sum of the second estimated click counts of the respective displayable information in the third time period, and the sum of the third estimated click counts of the respective displayable information in the third time period, obtain the initial value of the second click-through rate calibration coefficient of the rough sorting model;

[0049] Based on the sum of the second estimated conversion counts of the respective displayable information in the third time period, and the sum of the third estimated conversion counts of the respective displayable information in the third time period, obtain the initial value of the second conversion rate calibration coefficient of the rough sorting model.

[0050] Optionally, the predicted display result includes an estimated click-through rate and an estimated conversion rate; the calibration coefficient includes a click-through rate calibration coefficient and a conversion rate calibration coefficient;

[0051] The fourth acquisition subunit is configured to obtain the estimated click counts and estimated conversion counts of the respective displayable information corresponding to the refined sorting model in the fourth time period based on the actual exposure counts of the respective displayable information in the fourth time period, and the estimated click-through rate and estimated conversion rate of the respective displayable information corresponding to the refined sorting model in the fourth time period;

[0052] Based on the actual exposure counts of the respective displayable information in the fourth time period, and the estimated click-through rate and estimated conversion rate of the respective displayable information corresponding to the rough sorting model in the fourth time period, obtain the estimated click counts and estimated conversion counts of the respective displayable information corresponding to the rough sorting model in the fourth time period;

[0053] Based on the estimated click counts of the respective displayable information corresponding to the refined sorting model in the fourth time period, and the estimated click counts of the respective displayable information corresponding to the rough sorting model in the fourth time period, obtain the target values of the click-through rate calibration coefficients of the respective displayable information corresponding to the rough sorting model;

[0054] Based on the estimated conversion numbers of the respective displayable information in the fourth time period corresponding to the selected sorting model, and the estimated conversion numbers of the respective displayable information in the fourth time period corresponding to the rough sorting model, obtain the target values of the conversion rate calibration coefficients of the respective displayable information corresponding to the rough sorting model.

[0055] Optionally, the fourth obtaining unit is configured to perform weighted summation on the initial value of the second calibration coefficient and the target values of the calibration coefficients of the respective displayable information corresponding to the rough sorting model, to obtain the calibration coefficients of the respective displayable information corresponding to the rough sorting model.

[0056] On the other hand, a computer device is provided, which includes a processor and a memory. At least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the information display method as described above.

[0057] On the other hand, a computer-readable storage medium is provided. At least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the information display method as described above.

[0058] The technical solution provided by this application may include the following beneficial effects:

[0059] During the process of information display, the calibration coefficients of each sub-sorting model can be obtained through the actual display results of each displayable information and the predicted display results of the displayable information sorting model with cascaded sub-sorting models, so as to calibrate the displayable information sorting model. Based on the sorting results of the calibrated displayable information sorting model, each displayable information is displayed, realizing the update of the displayable information sorting model based on the prediction effect and the actual display effect of the cascaded multi-level displayable information sorting model, thereby improving the accuracy of the displayable information sorting model.

[0060] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0062] Figure 1Shows the working flowchart of an advertising trading platform with a multi-level sorting model shown in an exemplary embodiment of the present application;

[0063] Figure 2 Shows a schematic diagram of the relationship among an advertising trading platform, a media owner, and an advertiser shown in an exemplary embodiment of the present application;

[0064] Figure 3 Shows the flowchart of training a multi-level sorting model shown in an exemplary embodiment of the present application;

[0065] Figure 4 Is a schematic structural diagram of an information display system shown according to an exemplary embodiment;

[0066] Figure 5 Shows the flowchart of an information display method provided in an exemplary embodiment of the present application;

[0067] Figure 6 Shows the flowchart of an information display method provided in an exemplary embodiment of the present application;

[0068] Figure 7 Shows a schematic diagram of a real-time data stream multi-level calibration algorithm framework shown in an exemplary embodiment of the present application;

[0069] Figure 8 Shows the block diagram of an information display device provided in an exemplary embodiment of the present application;

[0070] Figure 9 Is a schematic block diagram of a computer device shown according to an exemplary embodiment. Detailed implementation

[0071] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0072] The embodiments of the present application propose an information display method. This solution can calibrate the sortable information model based on the actual display effect of the sortable information on the basis of the sortable information ranking model trained by artificial intelligence (AI), thereby improving the accuracy of the model. For ease of understanding, the terms related to the present application are explained below.

[0073] 1) Artificial intelligence

[0074] Artificial intelligence uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, including theories, methods, technologies, and application systems that can perceive the environment, acquire knowledge, and use knowledge to achieve the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling machines to have the functions of perception, reasoning, and decision-making.

[0075] Artificial intelligence technology is an interdisciplinary subject with a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0076] 2) Machine Learning (ML)

[0077] Machine learning is an interdisciplinary subject that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning from demonstration.

[0078] 3) Media owner

[0079] A media owner refers to an entity that owns an Internet platform, which can be a platform or an individual. For example, WeChat Moments, official accounts, news applications, electronic newspapers and magazines, etc. Generally, it has a large number of user visits (also known as user traffic). For media owners, they can convert user visits into cash income by inserting advertising positions in the platform.

[0080] 4) Advertiser

[0081] An advertiser refers to an entity that displays its advertisements through the advertising positions of an Internet platform. For example, a brand owner that displays advertisements in WeChat Moments, or an entity that publishes advertisements in the advertising section of an electronic newspaper or magazine, etc. By displaying its advertisements through the advertising positions of the Internet platform, it can convey information related to itself to attract users.

[0082] 5) Online advertising

[0083] Online advertising, also known as Internet advertising, refers to the advertisements placed on the advertising spaces of Internet platforms (such as WeChat Moments, official accounts, news applications, e-newspapers and magazines, etc.). Online advertising can include picture advertisements, text advertisements, keyword advertisements, ranking advertisements, video advertisements, etc.

[0084] 6) Advertising Exchange (ADX, Ad Exchange)

[0085] An advertising exchange is an entity that links media owners and advertisers. It places advertisers' advertisements on the advertising spaces provided by media owners. In order to accurately place advertisers' advertisements to the target audience, advertising exchanges generally collect users' information for user profiling, so as to conduct accurate advertising placement based on users' interests, geographical locations or other data.

[0086] 7) Click-Through Rate (CTR)

[0087] Click-Through Rate refers to the click-through rate of online advertising, that is, the actual number of clicks (Click) of the advertisement divided by the display volume (Show content) of the advertisement. It is an important indicator to measure the effect of online advertising.

[0088] Its calculation formula can be expressed as:

[0089]

[0090] 8) Lite Predict Click TthroughRate (LiteCTR)

[0091] Lite Predict Click TthroughRate is the probability that an online advertising system predicts an online advertisement will be clicked after it is placed in a certain situation. It is an important part of the ranking model. Under the multi-level ranking model, it represents the predicted click-through rate model in the rough sorting, and the complexity of the model is low.

[0092] 9) Predict Click Through Rate (pCTR)

[0093] Predict Click Through Rate is the probability that an online advertising system predicts an online advertisement will be clicked after it is placed in a certain situation. It is an important part of the ranking model. Under the multi-level ranking model, it represents the predicted click-through rate model in the refined sorting, and the complexity of the model is high.

[0094] 10) Conversion Rate (CVR)

[0095] The conversion rate is an indicator to measure the effectiveness of online advertising, which refers to the conversion ratio of users from clicking on an online advertisement to becoming a validly activated, registered, or paying user, that is, the actual number of conversions of the online advertisement divided by the number of clicks on the online advertisement.

[0096] 11) Lite Predict Conversion Rate (LiteCVR)

[0097] The Lite Predict Conversion Rate is the probability predicted by the online advertising system that an online advertisement will be converted after being clicked in a certain situation. It is an important part of the ranking model and represents the conversion rate model in the rough selection ranking under the multi-level ranking model, with a low model complexity.

[0098] 12) Predict Conversion Rate (pCVR)

[0099] The Predict Conversion Rate is the probability predicted by the online advertising system that an online advertisement will be converted after being clicked in a certain situation. It is an important part of the ranking model and represents the conversion rate model in the refined selection ranking under the multi-level ranking model, with a high model complexity.

[0100] 13) Cost Per Mille (CPM)

[0101] The Cost Per Mille refers to the cost that an advertiser needs to pay after an online advertisement is displayed to one thousand visiting users on the Internet platform. Its calculation formula is:

[0102] Cost Per Mille = (advertising cost / number of arrivals) × 1000;

[0103] Among them, the advertising cost / number of arrivals is usually expressed in the form of a percentage. The number of arrivals refers to the number of valid visiting users of the online advertisement. For example, if the cost paid by an advertiser for a certain online advertisement is 10,000 yuan and the number of arrivals is 5,000,000 people, then its Cost Per Mille is:

[0104] 10000 / 5000000 × 1000 = 2 (yuan);

[0105] 14) bid

[0106] The bid refers to the price at which an advertiser bids for an online advertisement. In oCPM (Optimized Cost Per Mille), it is generally the price for one conversion.

[0107] 15) Optimized Cost Per Mille (oCPM)

[0108] oCPM uses the same charging method as CPM, where the online advertising platform determines the value of each ad to the user. In this model, the advertiser sets the conversion goal and cost per unit of the ad, and the online advertising platform optimizes the ad placement according to the advertiser's settings to achieve the goal as efficiently as possible. The charge after every thousand impressions of the online ad is positively correlated with the real-time bid of the online ad. The real-time cost per thousand impressions of the ad is as follows:

[0109] Real-time CPM = bid × pCVR × pCTR;

[0110] 16) Multi-level sorting model

[0111] Due to the huge number of online ads, for engineering efficiency considerations, multiple (e.g., 2, including rough screening and refined screening) pCTR and pCVR models with different complexities are generally implemented to calculate CPM for gradually screening the online ads that best meet the interests of users, advertisers, and media owners. Among them:

[0112] Real-time CPM for rough screening sorting = bid × LiteCTR × LiteCVR;

[0113] Real-time CPM for refined screening sorting = bid × pCTR × pCVR;

[0114] When online ads are placed on the Internet of Things platform, they usually adopt a bidding method for placement. There are various bidding methods for competitive ads. Advertisers can choose to bid by exposure (per thousand impressions) (CPM, Cost Per Mille), bid by click (CPC, Cost Per Click), or also choose to bid by conversion (CPA, Cost Per Action). Different bidding methods have different applicable scenarios. For example, the mainstream bidding method for search ads is CPC. For mobile application (APP) ads, more attention is paid to the conversion cost. Therefore, the CPA method is beneficial for promotion information providers to control the conversion cost.

[0115] With the development of ad placement, in addition to CPA, bidding methods based on conversion, such as oCPM (Optimized Cost Per Mille) and oCPA (Optimized Cost Per Action), have gradually emerged. What remains unchanged is that CPA, oCPM, and oCPA are all based on conversion bidding. Obviously, in the process of ad placement based on conversion bidding, the ad trading platform needs to know the real conversion volume of the ad, which can be used not only for deduction and balance control but also for real-time optimization of the ad effect according to the actual conversion volume, so as to target the ad to more suitable people.

[0116] The current advertising trading platform sorts and filters the advertisements placed on the advertising spaces of the Internet platform through a multi-level sorting model. Please refer to Figure 1 , which shows the workflow diagram of the advertising trading platform with a multi-level sorting model shown in an exemplary embodiment of the present application. As Figure 1 shown, the workflow of the advertising trading platform includes:

[0117] Step 110, receiving a user request.

[0118] Among them, the user request refers to the request that the advertising trading platform receives for a user to access the Internet platform of the media owner, as well as the request of the media owner to display online advertisements. The advertising trading platform is used to connect the advertiser and the media owner. Please refer to Figure 2 , which shows the schematic diagram of the relationship among the advertising trading platform, the media owner, and the advertiser shown in an exemplary embodiment of the present application. As Figure 2 shown, the advertising trading platform 220 places the advertisements of the advertiser 210 on the advertising spaces of the media owner 230, and at the same time can collect the user information corresponding to the media owner, perform user profiling, and target the advertisements of the advertiser according to the different user profiles corresponding to different media owners.

[0119] Step 120, rough selection and sorting of advertisements.

[0120] When the advertising trading platform obtains a user request, it calculates the LiteCTR and LiteCVR values of each advertisement in the rough selection sorting model by combining the user information and the advertisement information, and calculates the rough selection real-time bid of each advertisement according to the LiteCTR and LiteCVR values, that is, the rough selection real-time CPM. The advertising trading platform sorts the advertisements according to the final rough selection real-time CPM, and selects the top N advertisements with the highest ranking and feeds them back to the refined sorting model, where N is a positive integer.

[0121] Step 130, refined selection and sorting of advertisements.

[0122] When the advertising trading platform obtains a user request, it calculates the pCTR and pCVR values of each advertisement in the refined sorting model by combining the user information and the advertisement information, and calculates the real-time bid of each advertisement according to the pCTR and pCVR values. The advertising trading platform sorts the N advertisements fed back by the rough sorting model according to the final refined real-time bid, that is, the refined real-time CPM, and selects the top M advertisements with the highest ranking and sends them to the media owner, where M < N and M is a positive integer.

[0123] Step 140, display of winning advertisements.

[0124] The media owner receives the advertising information provided by the advertising trading platform, that is, M advertisements screened by the rough sorting model and the refined sorting model of the advertising trading platform, and displays the advertisements on the advertising space of the media owner. When the displayed advertisements accumulate enough display times on the media owner, the advertising trading platform can charge the corresponding fees to the advertisers according to the display times.

[0125] Step 150, click and conversion data return.

[0126] The advertising trading platform collects the historical delivery records and the click and conversion records of the delivered advertisements. Conduct verification of the delivery system effect and the next round of iteration of the new model.

[0127] In the prior art, the above-mentioned advertising rough sorting and advertising refined sorting can be completed by a multi-level sorting model composed of a rough sorting model and a refined sorting model. This multi-level sorting model can be applied to the server of the advertising trading platform. The basic process of predicting CTR and CVR can be realized by using machine learning algorithms. Please refer to Figure 3 , which shows the flowchart of training the multi-level sorting model shown in an exemplary embodiment of the present application. As Figure 3 shown, this process may include the following steps:

[0128] Step 310, data processing.

[0129] Optionally, before data processing, the advertising trading platform needs to collect user information and obtain advertising information. This user information can be the user's behavior information on various media owner platforms and other Internet platforms, the user's personal attribute information, the user's smart device information, the information of the user clicking and converting advertisements, and so on.

[0130] Perform operations such as denoising and missing value filling on the collected user information. Use image processing algorithms, natural language processing algorithms, machine learning algorithms, etc. to extract the corresponding user interest features from the user information, and extract semantic features from the text and images of the advertisements, etc. Finally, convert the user's interest features and the semantic features of the advertisements into vector forms that can be processed by machine learning algorithms; combine the user's interest features and the semantic features of the advertisements to generate a binary tuple <X, y> for the user's behavior record on the advertisement, where X = (x1, x2,..., x m)It contains m features of users and advertisements, such as the attribute features, behavior features, user interest preference features extracted from behaviors of users, the attribute features, image features, text features of advertisements, etc. In the click-through rate prediction task, y ∈ {1, 0}, which is used to indicate whether the user clicks on the advertisement. That is, when the user clicks on the advertisement, y = 1, and when the user does not click on the advertisement or closes the advertisement, y = 0. In the conversion rate prediction task, y indicates whether the user has a conversion for the advertisement. Here, conversion means that the user becomes an effectively activated, registered, or paying user of the entity corresponding to the advertisement by clicking on the advertisement. When the user has a conversion for the advertisement, then y = 1. When the user does not have a conversion for the advertisement, that is, even if a user clicks on the advertisement but does not perform activation, registration, or payment behavior, then the user is still judged as not having a conversion for the advertisement, and y = 0.

[0131] Step 320, model training.

[0132] By processing the behavior records of users on advertisements, the advertisement trading platform generates a large number of pairs <X i , y i >, i = 1,..., n. Using machine learning models, such as logistic regression, random forest, gradient boosting tree, deep neural network and its variant algorithms, to find an objective function f(X) such that y = f(X). Since in reality, we do not know the specific form of f(X), generally, machine learning algorithms will find an optimal f(X) by solving the following optimization equation:

[0133]

[0134] where L(·) is the loss function, which is used to measure the difference between y and f(X). Generally, the loss function can be defined as the logarithmic loss function or the cross-entropy loss function, etc. k is the number of advertisements, and n i is the number of binary tuple samples of the i-th advertisement. Select the function that minimizes the solution of the above optimization equation as the objective function, that is, obtain a multi-level sorting model.

[0135] Step 330, model evaluation.

[0136] Also known as online evaluation, it evaluates the quality of the multi-level sorting model obtained in the training stage, including running performance, prediction accuracy, etc.

[0137] Step 340, model calibration.

[0138] Use the validation set of the model evaluation to calibrate the multi-level sorting model obtained in the training stage after evaluation, where the validation set is an offline data set.

[0139] Step 350, online deployment.

[0140] The multi-level sorting model that has undergone quality assessment and calibration is further deployed to the advertising trading platform to estimate the click-through rate and conversion rate of the combination of users and advertisements.

[0141] Since in the process of training the multi-level sorting model, when calibrating the model using the validation set of model evaluation for calibration, it is only applicable to calibrating a single-level sorting model through a pre-acquired validation data set during the training process, not applicable to the multi-level sorting model, and the model cannot be changed during the subsequent model application process, resulting in a relatively poor accuracy of the sorting model. To solve the above problems, the present application provides an information display method, which can be applied to an information display system.

[0142] Figure 4 It is a schematic structural diagram of an information display system shown according to an exemplary embodiment. The system includes: an information publisher terminal 420, an information display device 440, and a server 460.

[0143] The information publisher terminal 420 can be a terminal device with network access capabilities and user interface display and interaction functions. For example, the information publisher terminal 420 can be a PC (such as a laptop or a desktop computer, etc.), a smart phone, a tablet computer, an e-book reader, and so on.

[0144] The information display device 440 can be a computer device that includes or is externally connected to an information display platform. For example, the information display device 440 can be a mobile phone, a tablet computer, an e-book reader, smart glasses, a smart watch, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop, a desktop computer, a set-top box, a game console, an outdoor advertising display screen, and an in-vehicle advertising display screen, and so on.

[0145] The information publisher terminal 420 and the information display device 44 are respectively connected to the server 460 through a communication network. Optionally, the communication network is a wired network or a wireless network.

[0146] The server 460 is a single server, or consists of several servers, or is a virtualization platform, or is a cloud computing service center.

[0147] Among them, the server 460 can include a resource sales platform 460a, an order management platform 460b, and an information management platform 460c.

[0148] Among them, the resource sales platform 460a is used to interact with the information publisher terminal 420, provide the service of querying available information display resources to the information publisher terminal 420 according to user operations, and provide the service of locking / subscribing to information display resources to the information publisher terminal 420.

[0149] The order management platform 460b is used to store and maintain the orders for information display resources subscribed by information publishers. For example, create an order for an information publisher to request to subscribe to information display resources, or delete orders that have completed information display tasks (such as reaching the number of exposures) in existing orders, and so on.

[0150] The information management platform 460c is used to manage the display of information published by information publishers. For example, push the information published by information publishers to the corresponding information display platforms according to the types of information display resources subscribed by information publishers, and count the exposure situations of this information on each information display platform, and so on.

[0151] Optionally, the system may further include a management device ( Figure 4 not shown), and the management device is connected to the server 460 through a communication network. Optionally, the communication network is a wired network or a wireless network.

[0152] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is usually the Internet, but can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some of the links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0153] Please refer to Figure 5 , which shows a flowchart of an information display method provided by an exemplary embodiment of the present application. The information display method can be executed by a server, and the server can be Figure 4 the server shown in the figure. The method may include the following steps:

[0154] Step 510, obtaining the actual display results of each displayable information; the displayable information is the information selected for display by predicting the prediction probability of receiving a specified user operation through a displayable information sorting model, and the prediction probability is obtained through prediction; the displayable information sorting model includes at least two cascaded sub-sorting models, and the actual display result refers to the actual data of each displayable information received after being displayed on the display platform and receiving the specified user operation.

[0155] Among them, the displayable information refers to the information actively recommended by the terminal to the user. The displayable information can be manifested as information such as articles, advertisements, music, etc. In the embodiments of the present application, the above displayable information is taken as an advertisement as an example to illustrate the present application.

[0156] The cost per thousand impressions is used to represent the cost that the corresponding advertiser needs to pay after displaying the advertisement to one thousand users who access the Internet platform. When the advertisement is placed on the Internet of Things platform, it is usually placed through a bidding method. The higher the cost per thousand impressions of the advertisement, the higher its exposure rate on the Internet of Things platform. The cost per thousand impressions is positively correlated with the bidding price, predicted click-through rate, and predicted conversion rate. Therefore, the accuracy of the prediction of the predicted click-through rate and predicted conversion rate affects the calculation result of the cost per thousand impressions, and further affects the actual display result of the displayable information.

[0157] Among them, the actual display result of each displayable information refers to the actual data of the specified operations of the users of the display platform that each displayable information can receive during the actual display process of each displayable information on the display platform. These actual data can be the statistical results obtained by the display platform collecting the data information of each displayable information within a specified time period and statistically analyzing these data information.

[0158] Optionally, the displayable information sorting model can be a multi-level sorting model. The training and working processes of the multi-level sorting model can refer to Figure 1 and Figure 3 the relevant content in the description of the embodiments shown, which will not be elaborated here.

[0159] The displayable information sorting model includes at least two cascaded sub-sorting models. Here, cascading means that there is a hierarchical relationship between the sub-sorting models, that is, the previous sub-sorting model can affect the next sub-sorting model. It can also be understood that due to the change of the calibration target and output result of the previous sub-sorting model, the calibration target and output result of the next sub-sorting model also change.

[0160] Step 520, obtain the predicted display results corresponding to each displayable information for at least two sub-sorting models. The predicted display results are used to indicate the predicted probabilities that at least two sub-sorting models respectively predict that each displayable information will receive specified user operations.

[0161] Among them, the predicted display results corresponding to each displayable information for at least two sub-sorting models mean that after each displayable information is input into the displayable information sorting model, each sub-sorting model in the displayable information sorting model predicts the click-through rate and conversion rate of the displayable information, and predicts the sorting results of each displayable information based on the predicted click-through rate and conversion rate combined with the bidding price corresponding to each displayable information. Each sub-sorting model can set several displayable information with higher sorting results to be displayed on the corresponding Internet of Things platform, while other displayable information can not be displayed.

[0162] Step 530: Obtain the calibration coefficients corresponding to at least two sub-ranking models for each displayable information based on the actual display results of each displayable information and the predicted display results of at least two sub-ranking models corresponding to each displayable information.

[0163] In the embodiments of the present application, calibration refers to adjusting the parameters in each sub-ranking model with the calibration target of each sub-ranking model as the output target of each sub-ranking model, that is, generating calibration coefficients, so that the predicted results of the click-through rate and / or conversion rate of each sub-ranking model are as close as possible to the calibration target corresponding to each sub-ranking model.

[0164] Since there is a cascade relationship between each sub-ranking model and it has real-time performance, the calibration results corresponding to each sub-ranking model are calibration results generated according to the current real-time calibration target, and unified calibration of the sub-ranking models can be achieved.

[0165] Step 540: Correct the predicted probabilities of each displayable information by at least two sub-ranking models based on the calibration coefficients of at least two sub-ranking models.

[0166] Optionally, within a subsequent preset time period, after correcting the click-through rate and / or conversion rate output by each sub-ranking model according to the calibration coefficients corresponding to at least two sub-ranking models for each displayable information, obtain the predicted probabilities of the corrected displayable information, calculate the cost per thousand impressions of each displayable information based on the corrected predicted probabilities, and sort them to display each displayable information, thereby improving the display effect of each displayable information.

[0167] In summary, the information display method provided by the embodiments of the present application can, during the information display process, calibrate the displayable information ranking model through the actual display results of each displayable information and the predicted display results of each sub-ranking model in the cascaded displayable information ranking model, and the calibration coefficients of each sub-ranking model, and display each displayable information based on the ranking results of the calibrated displayable information ranking model, realizing the update of the displayable information ranking model based on the prediction effect and the actual display effect of the cascaded multi-level displayable information ranking model, thereby improving the accuracy of the displayable information ranking model and further improving the display effect of each displayable information.

[0168] Please refer to Figure 6 , which shows a flowchart of an information display method provided by an exemplary embodiment of the present application. This information display method can be executed by a server, and the server can be the Figure 4 server shown. This method may include the following steps:

[0169] Step 610: Obtain the actual display results of each displayable information. The displayable information is the information selected for display by predicting the probability of receiving a specified user operation through a displayable information ranking model, and the predicted probability is obtained. The displayable information ranking model includes at least two cascaded sub-ranking models. The actual display result refers to the actual data of receiving a specified user operation after each displayable information is displayed on the display platform.

[0170] Optionally, the at least two sub-ranking models include a rough selection ranking model and a refined selection ranking model cascaded after the rough selection ranking model.

[0171] In the embodiments of the present application, the information display method provided by the present application is described by taking the displayable information ranking model as being composed of a rough selection ranking model and a refined selection ranking model.

[0172] Optionally, the actual display results may include the actual exposure number, the actual click number, and the actual conversion number.

[0173] Step 620: Obtain the predicted display results corresponding to each displayable information for at least two sub-ranking models. The predicted display results are used to indicate the predicted probabilities of at least two sub-ranking models respectively predicting receiving a specified user operation for each displayable information.

[0174] Optionally, the predicted display results may include the estimated click-through rate and the estimated conversion rate. The estimated click-through rate and the estimated conversion rate may be the estimated click-through rate and the estimated conversion rate corresponding to each sub-ranking model respectively.

[0175] Step 630: Based on the actual display results of each displayable information and the predicted display results corresponding to each displayable information for the refined selection ranking model, obtain the calibration coefficients corresponding to each displayable information for the refined selection ranking model.

[0176] Optionally, the calibration coefficients include a click-through rate calibration coefficient and a conversion rate calibration coefficient.

[0177] Optionally, the process of obtaining the calibration coefficients corresponding to each displayable information for the refined selection ranking model can be implemented as:

[0178] Step 631: Based on the actual display results of each displayable information and the predicted display results corresponding to each displayable information for the refined selection ranking model, obtain the initial value of the first calibration coefficient of the refined selection ranking model and the target value of the calibration coefficient corresponding to each displayable information for the refined selection ranking model.

[0179] Since the refined selection ranking model acts after the rough selection ranking model, for the current rough selection ranking model, the target value of its calibration coefficient is the ranking result of the refined selection ranking model in the previous time period. Please refer toFigure 7 , which shows a schematic diagram of the multi-level calibration algorithm framework for real-time data streams shown in an exemplary embodiment of the present application. As Figure 7 shown, taking advertising as an example, after the advertising trading platform receives a user request (S71), it first performs a rough selection and sorting of advertisements by combining user information and advertising information (S72), selects the top N advertisements with the highest rough selection real-time CPM among them, and performs a refined selection and sorting on them (S73), selects the top M advertisements with the highest refined selection real-time CPM among them and sends them to the media owner. After the media owner pulls the advertisement (S74), the advertisement is exposed (S75), and clicks (S76) and conversions (S77) performed by the user on the advertisement are responded to.

[0180] The rough selection and sorting are completed by the rough selection and sorting model, and the refined selection and sorting are completed by the refined selection and sorting model. When calibrating the refined selection and sorting model, the target value of the calibration coefficient is calculated in real time (S78) according to at least one of the real-time exposure rate, click-through rate, and conversion rate of the advertisement. When calibrating the rough selection and sorting model, the sorting result of the refined selection and sorting model in the previous time period is used as the target value of the calibration coefficient for calibration. Since the target value of the calibration coefficient of the rough selection and sorting model is the estimated result of the refined selection and sorting model, the estimated results of the rough selection and sorting model and the refined selection and sorting model will eventually tend to be the same, that is, the sorting results of the rough selection and sorting model for each advertisement will tend to be the same as the sorting results of the refined selection and sorting model for the advertisements fed back by the rough selection and sorting model, so as to achieve the purpose of joint calibration.

[0181] Optionally, the first initial value of the calibration coefficient of the refined selection and sorting model can be obtained based on the actual display results of each displayable information in the first time period and the predicted display results of each displayable information corresponding to the refined selection and sorting model in the first time period.

[0182] Among them, the process of obtaining the first initial value of the calibration coefficient of the refined selection and sorting model is the process of initializing the predicted display results of the displayable information sorting model. Since the predicted display results corresponding to each displayable information are values that change with time or the content of the media owner platform, therefore, in order to implement the calibration algorithm of the refined selection and sorting model, it is necessary to first initialize the predicted display results corresponding to each displayable information.

[0183] Optionally, the above process of obtaining the first initial value of the calibration coefficient can be implemented as:

[0184] S6311, obtain the sum of the actual click counts of each displayable information in the first time period, and the sum of the actual conversion counts of each displayable information in the first time period.

[0185] Optionally, the first time period can be the nearest K hours from the time point when the sum of the click counts is obtained, and K is a positive number.

[0186] S6312. Based on the actual exposure counts of each displayable information within the first time period, and the estimated click-through rate and estimated conversion rate of each displayable information corresponding to the selected sorting model within the first time period, obtain the sum of the first estimated click counts of each displayable information corresponding to the selected sorting model within the first time period, and the sum of the first estimated conversion counts of each displayable information within the first time period.

[0187] Among them, the estimated click-through rate refers to the probability that a displayable information is estimated to be clicked after being placed in a certain situation by the displayable information sorting model, and the estimated conversion rate refers to the probability that a displayable information is estimated to be converted after being placed in a certain situation by the displayable information sorting model.

[0188] Optionally, a first sampling period within the first time period can be preset, and the estimated click-through rate and estimated conversion count of each displayable information are obtained according to this first sampling period, so as to calculate the total estimated click count and total estimated conversion count of each displayable information within the first time period.

[0189] For example, if the preset sampling period within K hours is 5 minutes, then the estimated click-through rate and estimated conversion rate of each message are obtained every 5 minutes, and the estimated click count and estimated conversion count within 5 minutes are calculated, so as to calculate the total estimated click count and total estimated conversion count within K hours.

[0190] It should be noted that this sampling period can be set according to actual needs, and this application does not limit it.

[0191] S6313. Based on the sum of the actual click counts of each displayable information within the first time period and the sum of the first estimated click counts, obtain the initial value of the first click-through rate calibration coefficient of the selected sorting model.

[0192] Since in the calculation process of the estimated click-through rate and the click-through rate, the estimated click-through rate is calculated by dividing the estimated click count by the actual exposure count, and the click-through rate is calculated by dividing the actual click count by the actual exposure count. Therefore, when calculating the initial value of the first click-through rate calibration coefficient, when dividing the total actual click-through rate by the total estimated click-through rate, the actual exposure count can be reduced. Therefore, calculating the initial value of the first click-through rate calibration coefficient can be expressed as the sum of the actual click counts within the first time period divided by the sum of the estimated click counts within the first time period, and its calculation formula can be expressed as:

[0193]

[0194] Among them, fix_pctr_ratio(0) represents the initial value of the first click-through rate calibration coefficient, ∑click i (0) represents the sum of the actual click counts within the first time period, ∑pctri (0) represents the sum of the estimated click counts in the first time period, and i represents the i-th displayable information.

[0195] S6314. Obtain the initial value of the first conversion rate calibration coefficient of the refined ranking model based on the sum of the actual conversion counts of each displayable information in the first time period and the sum of the first estimated conversion counts.

[0196] Since in the calculation process of the estimated conversion rate and the conversion rate, the estimated conversion rate is obtained by dividing the estimated conversion count by the actual click count, and the conversion rate is obtained by dividing the actual conversion count by the actual click count. Therefore, when calculating the initial value of the first conversion rate calibration coefficient, when dividing the total actual conversion rate by the total estimated conversion rate, the actual click count can be reduced. Therefore, calculating the initial value of the first conversion rate calibration coefficient can be expressed as the sum of the actual conversion counts in the first time period divided by the sum of the estimated conversion counts in the first time period, and its calculation formula can be expressed as:

[0197]

[0198] Among them, fix_pcvr_ratio(0) represents the initial value of the first conversion rate calibration coefficient, ∑conv i (0) represents the sum of the actual conversion counts in the first time period, ∑pcvr i (0) represents the sum of the estimated conversion counts in the first time period, and i represents the i-th displayable information.

[0199] Optionally, based on the actual display results of each displayable information in the second time period, obtain the target value of the calibration coefficient corresponding to each displayable information for the refined ranking model.

[0200] Optionally, the second time period can be in minutes to ensure the real-time performance of the refined ranking model calibration algorithm. For example, the first time period can be the most recent 1 minute.

[0201] Optionally, the target value of the calibration coefficient of the refined ranking model can include at least one of the exposure rate, click-through rate, and conversion rate of the displayable information;

[0202] Among them, the exposure rate of the displayable information refers to the probability that the displayable information is shown to the media owner users; the click-through rate refers to the proportion of users who click on the displayable information among the users who receive the push of a certain promotion information; the conversion rate refers to the proportion of users who perform a specific behavior on the displayable information among the users who click on a certain displayable information. This specific behavior can be set according to the actual situation. For example, if the displayable information is an article, then the user's clicking and reading the article can be used as the corresponding specific behavior, and the user's clicking and reading the article is recorded as one conversion; if the displayable information is an advertisement of a mobile application APP, the specific behavior can be set as the behavior of the user downloading the APP, and each time the APP is downloaded, it is recorded as one conversion.

[0203] For example, within a certain period of time, there are 1,000 users of a certain media owner. The advertising trading platform pushes a certain displayable information to one of the end users, which is equivalent to the displayable information being exposed once. Assuming that 100 users receive the displayable information within this period of time, then the exposure rate of the displayable information is 10%. Assuming that 50 out of the 100 end users who view the displayable information click on the displayable information, then the click-through rate of the displayable information is 50%. Assuming that among the 50 end users who click on the displayable information, 10 end users perform a specific behavior on the displayable information, then the conversion rate of the displayable information is 20%.

[0204] In the embodiment of the present application, the information display method provided by the present application is described by taking the target values of the calibration coefficients of the refined sorting model as the click-through rate and the conversion rate.

[0205] Optionally, the process of obtaining the target values of the calibration coefficients of the refined sorting model corresponding to each displayable information can be implemented as follows:

[0206] S6315, obtain the actual click count of each displayable information within the second time period, and the actual conversion count of each displayable information within the second time period.

[0207] Optionally, a second sampling period within the second time period can be set in advance, and according to this second sampling period, obtain the actual click count and the actual conversion count of each displayable information within each second sampling period within the second time period, so as to calculate the actual click count of each displayable information within the second time period and the actual conversion count within the second time period. Its calculation formula can be expressed as:

[0208] sum_click(t) = ∑click i (t)

[0209] sum_conv(t) = ∑conv i (t)

[0210] where \(t\) represents the second time period, \(sum\_click(t)\) represents the actual number of clicks within the second time period, and \(click\) i (t) represents the actual number of conversions within each second sampling period.

[0211] S6316. Based on the actual number of impressions of each displayable information within the second time period, as well as the estimated click-through rate and estimated conversion rate of each displayable information corresponding to the refined sorting model within the second time period, obtain the estimated number of clicks and estimated number of conversions of each displayable information corresponding to the refined sorting model within the second time period.

[0212] Optionally, obtain the estimated click-through rate and estimated conversion rate of each displayable information within each second sampling period of the second time period according to the second sampling period, so as to combine the actual number of impressions of each displayable information within each second sampling period of the second time period, calculate the estimated number of clicks and estimated number of conversions within each second sampling period, and thus calculate the estimated number of clicks and estimated number of conversions of each displayable information within the second time period. Its calculation formula can be expressed as:

[0213] \(sum\_pctr(t)=\sum pctr\) i (t)

[0214] \(sum\_pcvr(t)=\sum pcvr\) i (t)

[0215] where \(t\) represents the second time period, \(sum\_pctr(t)\) represents the estimated number of clicks within the second time period, \(pctr\) i (t) represents the estimated number of clicks within each second sampling period, \(sum\_pcvr(t)\) represents the estimated number of conversions within the second time period, and \(pcvr\) i (t) represents the estimated number of conversions within each second sampling period.

[0216] S6317. Based on the actual number of clicks of each displayable information within the second time period, and the estimated number of clicks of each displayable information corresponding to the refined sorting model within the second time period, obtain the target value of the click-through rate calibration coefficient of each displayable information corresponding to the refined sorting model. Its calculation formula can be expressed as:

[0217]

[0218] where \(t\) represents the second time period, \(update\_pctr\_ratio\) i (t) represents the target value of the click-through rate calibration coefficient of the refined sorting model.

[0219] S6318. Obtain the target value of the conversion rate calibration coefficient of each displayable information corresponding to the selected sorting model based on the actual conversion number of each displayable information in the second time period and the estimated conversion number of each displayable information in the second time period corresponding to the selected sorting model. Its calculation formula can be expressed as:

[0220]

[0221] Among them, t represents the second time period, and update_pcvr_ratio i (t) represents the target value of the conversion rate calibration coefficient of the selected sorting model.

[0222] Step 632. Obtain the calibration coefficient of each displayable information corresponding to the selected sorting model based on the initial value of the first calibration coefficient and the target value of the calibration coefficient of each displayable information corresponding to the selected sorting model.

[0223] Optionally, perform weighted summation on the initial value of the first calibration coefficient and the target value of the calibration coefficient of each displayable information corresponding to the selected sorting model to obtain the calibration coefficient of each displayable information corresponding to the selected sorting model. Its calculation formula can be expressed as:

[0224] fix_pctr_ratio i (t + 1) = (1 - α) * fix_pctr_ratio i (0) + α * update_pctr_ratio i (t)

[0225] fix_pcvr_ratio i (t + 1) = (1 - α) * fix_pcvr_ratio i (t) + α * update_pcvr_ratio i (t)

[0226] Among them, t + 1 represents the next time period after the second time period, and fix_pctr_ratio i (t + 1) and fix_pcvr_ratio i (t + 1) represent the calibration coefficient of the selected sorting model, and α represents the smoothing coefficient, which is used to balance the initial value of the first click-through rate calibration coefficient and the click-through rate calibration coefficient target value of the selected sorting model, and to balance the initial value of the first conversion rate calibration coefficient and the conversion rate calibration coefficient target value of the selected sorting model.

[0227] Generally speaking, the smoothing coefficient α represents the response speed of the exponential smoothing model to the changes in the time series and determines the ability of the prediction model to smooth the random errors. The value of α can be changed according to business requirements. Its value is related to the number of samples in the first period. Generally, the more samples there are, the larger the value of α that can be set.

[0228] Step 640: Based on the predicted display results of the refined ranking model corresponding to each displayable information and the predicted display results of the rough ranking model corresponding to each displayable information, obtain the calibration coefficient of the rough ranking model corresponding to each displayable information.

[0229] Optionally, the process of obtaining the calibration coefficient of the rough ranking model corresponding to each displayable information can be implemented as follows:

[0230] Step 641: Based on the predicted display results of the refined ranking model corresponding to each displayable information and the predicted display results of the rough ranking model corresponding to each displayable information, obtain the initial value of the second calibration coefficient of the rough ranking model and the target value of the calibration coefficient of the initial ranking model corresponding to each displayable information.

[0231] Optionally, the initial value of the second calibration coefficient of the rough ranking model can be obtained based on the predicted display results of the refined ranking model corresponding to each displayable information in the third period and the predicted display results of the rough ranking model corresponding to each displayable information in the third period.

[0232] Optionally, the third period can be the same period as the first period.

[0233] Optionally, the process of obtaining the initial value of the second calibration coefficient of the rough ranking model can be implemented as follows:

[0234] S6411: Based on the actual exposure numbers of each displayable information in the third period and the estimated click-through rates and estimated conversion rates of the refined ranking model corresponding to each displayable information in the third period, obtain the sum of the second estimated click numbers of each displayable information corresponding to the refined ranking model in the third period and the sum of the second estimated conversion numbers of each displayable information in the third period.

[0235] S6412: Based on the actual exposure numbers of each displayable information in the third period and the estimated click-through rates and estimated conversion rates of the rough ranking model corresponding to each displayable information in the third period, obtain the sum of the third estimated click numbers of each displayable information corresponding to the rough ranking model in the third period and the sum of the third estimated conversion numbers of each displayable information in the third period.

[0236] Optionally, the estimated click-through rate of the rough selection ranking model can be referred to as the lightweight estimated click-through rate, and the estimated conversion rate of the rough selection ranking model can be referred to as the lightweight estimated conversion rate.

[0237] Optionally, the third sampling period within the third time period can be preset, and based on this third sampling period, the lightweight estimated click-through rate and lightweight estimated conversion rate of each displayable information are obtained to calculate the total lightweight estimated click count and total lightweight estimated conversion count of each displayable information within the third time period. When the third time period is the same as the first time period, the third sampling period should also be consistent with the first sampling period.

[0238] S6413. Based on the sum of the second estimated click counts of each displayable information within the third time period and the sum of the third estimated click counts of each displayable information within the third time period, obtain the initial value of the second click-through rate calibration coefficient of the rough selection ranking model.

[0239] Since in the calculation process of the lightweight estimated click-through rate and the estimated click-through rate, the lightweight estimated click-through rate is calculated by dividing the lightweight estimated click count (the third estimated click count) by the actual exposure count, and the estimated click-through rate is calculated by dividing the estimated click count by the actual exposure count. When calculating the initial value of the second click-through rate calibration coefficient, when dividing the total estimated click-through rate by the total lightweight estimated click-through rate, the actual exposure count can be reduced. Therefore, calculating the initial value of the second click-through rate calibration coefficient can be expressed as the sum of the estimated click counts within the third time period divided by the sum of the lightweight estimated click counts (the third estimated click counts) within the third time period, and its calculation formula can be expressed as:

[0240]

[0241] Among them, fix_litectr_ratio(0) represents the initial value of the second click-through rate calibration coefficient, ∑pctr i (0) represents the sum of the second estimated click counts within the third time period, ∑litectr i (0) represents the sum of the third estimated click counts within the third time period, and i represents the i-th displayable information.

[0242] S6414. Based on the sum of the second estimated conversion counts of each displayable information within the third time period and the sum of the third estimated conversion counts of each displayable information within the third time period, obtain the initial value of the second conversion rate calibration coefficient of the rough selection ranking model.

[0243] Since in the calculation processes of the lightweight estimated conversion rate and the estimated conversion rate, the lightweight estimated conversion rate is obtained by dividing the lightweight estimated conversion number (the third estimated conversion number) by the actual click number, and the estimated conversion rate is obtained by dividing the estimated conversion number by the actual click number. When calculating the initial value of the second conversion rate calibration coefficient, when dividing the total estimated conversion rate by the total lightweight estimated conversion rate, the actual click number can be reduced. Therefore, the calculation of the initial value of the second conversion rate calibration coefficient can be expressed as the sum of the second estimated conversion numbers in the third time period divided by the sum of the third estimated conversion numbers in the third time period, and its calculation formula can be expressed as:

[0244]

[0245] Among them, fix_litecvr_ratio(0) represents the initial value of the second conversion rate calibration coefficient, and ∑pcvr i (0) represents the sum of the third estimated conversion numbers in the third time period, and i represents the i-th displayable information.

[0246] Optionally, the server can obtain the calibration coefficient target values of the rough sorting model corresponding to each displayable information based on the predicted display results of the refined sorting model corresponding to each displayable information in the fourth time period.

[0247] Optionally, the fourth time period can be the same time period as the second time period.

[0248] Optionally, the process of obtaining the calibration coefficient target values of the rough sorting model corresponding to each displayable information can be implemented as:

[0249] S6415, based on the actual exposure numbers of each displayable information in the fourth time period, and the estimated click-through rate and estimated conversion rate of the refined sorting model corresponding to each displayable information in the fourth time period, obtain the estimated click numbers and estimated conversion numbers of each displayable information corresponding to the refined sorting model in the fourth time period.

[0250] Optionally, the fourth sampling period in the fourth time period can be set in advance, and the estimated click-through rate and estimated conversion rate of the refined sorting model of each displayable information in each fourth sampling period in the fourth time period are obtained according to this fourth sampling period, so as to combine the actual exposure numbers of each displayable information in each fourth sampling period in the fourth time period, thereby calculating the estimated click numbers of each displayable information in the fourth time period and the estimated conversion numbers in the fourth time period. When the fourth time period is the same time period as the second time period, the fourth sampling period should also be consistent with the second sampling period.

[0251] When the fourth time period is the same time period as the second time period, its calculation formula can be expressed as:

[0252] sum_pctr i (t) = ∑pctr i (t)

[0253] sum_pcvr i (t) = ∑pcvr i (t)

[0254] Wherein, t represents the fourth time period, and sum_pctr i (t) represents the estimated number of clicks of the refined sorting model within the fourth time period, and pctr i (t) represents the estimated number of conversions of the refined sorting model within each sampling period.

[0255] S6416. Based on the actual exposure numbers of each displayable information within the fourth time period, as well as the estimated click-through rate and estimated conversion rate of each displayable information corresponding to the rough sorting model within the fourth time period, obtain the estimated number of clicks and estimated number of conversions of each displayable information corresponding to the rough sorting model within the fourth time period.

[0256] Optionally, obtain the lightweight estimated click-through rate and lightweight estimated conversion rate of each displayable information within each fourth sampling period of the fourth time period according to the fourth sampling period, so as to combine the actual exposure numbers of each displayable information within each fourth sampling period of the fourth time period, calculate the lightweight estimated number of clicks and lightweight estimated number of conversions within each fourth sampling period, and thus calculate the lightweight estimated number of clicks and lightweight estimated number of conversions of each displayable information within the fourth time period. Its calculation formula can be expressed as:

[0257] sum_litectr i (t) = ∑litectr i (t)

[0258] sum_litecvr i (t) = ∑litecvr i (t)

[0259] Wherein, t represents the fourth time period, sum_pctr(t) represents the lightweight estimated number of conversions within the fourth time period, and pctr i (t) represents the lightweight estimated number of conversions within each fourth sampling period, sum_pcvr(t) represents the lightweight estimated number of conversions within the fourth time period, and pcvr i (t) represents the lightweight estimated number of conversions within each fourth sampling period.

[0260] S6417. Based on the estimated number of clicks of each displayable information in the fourth time period corresponding to the refined sorting model and the estimated number of clicks of each displayable information in the fourth time period corresponding to the rough sorting model, obtain the target value of the click-through rate calibration coefficient of each displayable information corresponding to the rough sorting model respectively. Its calculation formula can be expressed as:

[0261]

[0262] Among them, t represents the fourth time period, and update_litectr_ratio(t) represents the target value of the click-through rate calibration coefficient of the rough sorting model.

[0263] S6418. Based on the estimated number of conversions of each displayable information in the fourth time period corresponding to the refined sorting model and the estimated number of conversions of each displayable information in the fourth time period corresponding to the rough sorting model, obtain the target value of the conversion rate calibration coefficient of each displayable information corresponding to the rough sorting model respectively. Its calculation formula can be expressed as:

[0264]

[0265] Among them, t represents the fourth time period, and update_litecvr_ratio(t) represents the target value of the conversion rate calibration coefficient of the rough sorting model.

[0266] Step 642. Based on the second calibration coefficient initial value and the target value of the calibration coefficient of each displayable information corresponding to the initial sorting model respectively, obtain the calibration coefficient of each displayable information corresponding to the rough sorting model.

[0267] Optionally, perform weighted summation on the second calibration coefficient initial value and the target value of the calibration coefficient of each displayable information corresponding to the rough sorting model respectively to obtain the calibration coefficient of each displayable information corresponding to the rough sorting model. Its calculation formula can be expressed as:

[0268] fix_litectr_ratio i (t + 1) = (1 - α) * fix_litectr_ratio i (t) + α * update_litectr_ratio i (t)

[0269] fix_litecvr_ratio i (t + 1) = (1 - α) * fix_litecvr_ratio i (t) + α * update_litecvr_ratio i (t)

[0270] where t + 1 represents the next time period after the fourth time period, and fix_pctr_ratio i (t + 1) and fix_pcvr_ratio i (t + 1) represents the calibration coefficient of the rough sorting model, α represents the smoothing coefficient, which is used to balance the initial value of the second click-through rate calibration coefficient and the target value of the click-through rate calibration coefficient of the rough sorting model, and to balance the initial value of the first conversion rate calibration coefficient and the target value of the conversion rate calibration coefficient of the rough sorting model.

[0271] Step 650: Based on the calibration coefficients of at least two sub-sorting models, correct the prediction probabilities of at least two sub-sorting models for each displayable information in the future.

[0272] Optionally, for the refined sorting model of each displayable information sorting model, after calibration by combining its corresponding calibration coefficient, the formula for calculating the real-time cost per thousand impressions is expressed as:

[0273] Refined sorting real-time CPM = bid × pCVR × pCTR * fix_pctr_ratio i (t + 1) * fix_pcvr_ratio i (t + 1)

[0274] where pCVR × pCTR * fix_pctr_ratio i (t + 1) * fix_pcvr_ratio i (t + 1) represents the prediction probability of the refined sorting model for each displayable information after correction. Based on the corrected prediction probability, calculate the real-time cost per thousand impressions of each displayable information, so as to re-sort each displayable information, and further determine the display method of each displayable information.

[0275] For the rough sorting model of each displayable information sorting model, after calibration by combining its corresponding calibration coefficient, the formula for calculating the real-time cost per thousand impressions is expressed as:

[0276] Rough sorting real-time CPM = bid * LiteCVR * LiteCTR * fix_litectr_ratio i (t + 1) * fix_litecvr_ratio i (t + 1)

[0277] where LiteCVR * LiteCTR * fix_litectr_ratio i (t + 1) * fix_litecvr_ratio i(t + 1) represents the predicted probability of the corrected rough selection ranking model for each displayable information. Based on the corrected predicted probability, the real-time cost per thousand impressions of each displayable information for rough selection ranking is calculated, so as to re-rank each displayable information, and further determine the display mode of each displayable information.

[0278] Optionally, there are differences between the calibration coefficients of the refined ranking models corresponding to different displayable information, and there are differences between the calibration coefficients of the rough selection ranking models corresponding to different displayable information.

[0279] In the actual application process of the displayable information ranking model, the calibration processes of the refined ranking model and the rough selection ranking model are carried out in real time according to real-time data, so as to ensure the accuracy of the displayable information ranking model.

[0280] In summary, the information display method provided by the embodiments of the present application can obtain the calibration coefficients of each sub-ranking model through the actual display results of each displayable information and the predicted display results of the cascade sub-ranking models of the displayable information ranking model during the information display process, so as to calibrate the displayable information ranking model. Based on the ranking results of the calibrated displayable information ranking model, each displayable information is displayed, realizing the update of the displayable information ranking model based on the prediction effect and the actual display effect of the cascade multi-level displayable information ranking model, thereby improving the accuracy of the displayable information ranking model, and further improving the display effect of each displayable information.

[0281] Please refer to Figure 8 , which shows a block diagram of an information display device provided by an exemplary embodiment of the present application. This information display device can be applied to a server, and the server can be implemented as Figure 1 the server shown, and the device may include:

[0282] A first acquisition module 810, configured to acquire the actual display results of each displayable information; the displayable information is the information selected for display by predicting the probability of receiving a specified user operation through a displayable information ranking model; the displayable information ranking model includes at least two cascaded sub-ranking models, and the actual display result refers to the actual data of receiving a specified user operation after each displayable information is displayed on the display platform;

[0283] A second acquisition module 820, configured to acquire the predicted display results of each displayable information corresponding to at least two sub-ranking models, and the predicted display results are used to indicate the predicted probabilities of at least two sub-ranking models respectively predicting receiving a specified user operation for each displayable information;

[0284] The third acquisition module 830 is configured to obtain calibration coefficients corresponding to at least two sub-ranking models for each displayable information based on the actual display results of each displayable information and the predicted display results of at least two sub-ranking models corresponding to each displayable information.

[0285] The display module 840 is configured to correct the predicted probabilities of at least two sub-ranking models for each displayable information subsequently based on the calibration coefficients of the at least two sub-ranking models.

[0286] Optionally, the at least two sub-ranking models include a rough selection ranking model and a refined selection ranking model cascaded after the rough selection ranking model.

[0287] The second acquisition module 820 includes:

[0288] The first acquisition sub-module is configured to obtain calibration coefficients corresponding to the refined selection ranking model for each displayable information based on the actual display results of each displayable information and the predicted display results of the refined selection ranking model corresponding to each displayable information.

[0289] The second acquisition sub-module is configured to obtain calibration coefficients corresponding to the rough selection ranking model for each displayable information based on the predicted display results of the refined selection ranking model corresponding to each displayable information and the predicted display results of the rough selection ranking model corresponding to each displayable information.

[0290] Optionally, the first acquisition sub-module includes:

[0291] The first acquisition unit is configured to obtain an initial value of the first calibration coefficient of the refined selection ranking model and calibration coefficient target values corresponding to the refined selection ranking model for each displayable information based on the actual display results of each displayable information and the predicted display results of the refined selection ranking model corresponding to each displayable information.

[0292] The second acquisition unit is configured to obtain calibration coefficients corresponding to the refined selection ranking model for each displayable information based on the initial value of the first calibration coefficient and the calibration coefficient target values corresponding to the refined selection ranking model for each displayable information.

[0293] Optionally, the first acquisition unit includes:

[0294] The first acquisition sub-unit is configured to obtain an initial value of the first calibration coefficient of the refined selection ranking model based on the actual display results of each displayable information in the first time period and the predicted display results of the refined selection ranking model corresponding to each displayable information in the first time period.

[0295] A second obtaining subunit, configured to obtain, based on the actual display results of each displayable information in the second time period, the target values of the calibration coefficients of each displayable information corresponding to the refined sorting model.

[0296] Optionally, the actual display results include the actual number of exposures, the actual number of clicks, and the actual number of conversions; the predicted display results include the predicted click-through rate and the predicted conversion rate; the calibration coefficients include the click-through rate calibration coefficient and the conversion rate calibration coefficient.

[0297] This first obtaining subunit is configured to:

[0298] Obtain the sum of the actual number of clicks of each displayable information in the first time period, and the sum of the actual number of conversions of each displayable information in the first time period.

[0299] Based on the actual number of exposures of each displayable information in the first time period, and the predicted click-through rate and the predicted conversion rate of each displayable information corresponding to the refined sorting model in the first time period, obtain the sum of the first predicted number of clicks of each displayable information corresponding to the refined sorting model in the first time period, and the sum of the first predicted number of conversions of each displayable information in the first time period.

[0300] Based on the sum of the actual number of clicks of each displayable information in the first time period, and the sum of the first predicted number of clicks, obtain the initial value of the first click-through rate calibration coefficient of the refined sorting model.

[0301] Based on the sum of the actual number of conversions of each displayable information in the first time period, and the sum of the first predicted number of conversions, obtain the initial value of the first conversion rate calibration coefficient of the refined sorting model.

[0302] Optionally, the actual display results include the actual number of exposures, the actual number of clicks, and the actual number of conversions; the predicted display results include the predicted click-through rate and the predicted conversion rate; the calibration coefficients include the click-through rate calibration coefficient and the conversion rate calibration coefficient.

[0303] This second obtaining subunit is configured to:

[0304] Obtain the actual number of clicks of each displayable information in the second time period, and the actual number of conversions of each displayable information in the second time period.

[0305] Based on the actual number of exposures of each displayable information in the second time period, and the predicted click-through rate and the predicted conversion rate of each displayable information corresponding to the refined sorting model in the second time period, obtain the predicted number of clicks and the predicted number of conversions of each displayable information corresponding to the refined sorting model in the second time period.

[0306] Obtain the target value of the click-through rate calibration coefficient corresponding to each displayable information for the refined sorting model based on the actual number of clicks of each displayable information in the second time period and the predicted number of clicks of each displayable information in the second time period corresponding to the refined sorting model;

[0307] Obtain the target value of the conversion rate calibration coefficient corresponding to each displayable information for the refined sorting model based on the actual number of conversions of each displayable information in the second time period and the predicted number of conversions of each displayable information in the second time period corresponding to the refined sorting model.

[0308] Optionally, the second obtaining unit is used to perform a weighted sum of the initial value of the first calibration coefficient and the target value of the calibration coefficient corresponding to each displayable information for the refined sorting model to obtain the calibration coefficient corresponding to each displayable information for the refined sorting model.

[0309] Optionally, the second obtaining sub-module includes:

[0310] The third obtaining unit is used to obtain the initial value of the second calibration coefficient of the rough sorting model and the target value of the calibration coefficient corresponding to each displayable information for the initial sorting model based on the predicted display results corresponding to each displayable information for the refined sorting model and the predicted display results corresponding to each displayable information for the rough sorting model;

[0311] The fourth obtaining unit is used to obtain the calibration coefficient corresponding to each displayable information for the rough sorting model based on the initial value of the second calibration coefficient and the target value of the calibration coefficient corresponding to each displayable information for the initial sorting model.

[0312] Optionally, the third obtaining unit includes:

[0313] The third obtaining sub-unit is used to obtain the initial value of the second calibration coefficient of the rough sorting model based on the predicted display results corresponding to each displayable information for the refined sorting model in the third time period and the predicted display results corresponding to each displayable information for the rough sorting model in the third time period;

[0314] The fourth obtaining sub-unit is used to obtain the target value of the calibration coefficient corresponding to each displayable information for the rough sorting model based on the predicted display results corresponding to each displayable information for the refined sorting model in the fourth time period.

[0315] Optionally, the predicted display results include the predicted click-through rate and the predicted conversion rate; the calibration coefficients include the click-through rate calibration coefficient and the conversion rate calibration coefficient;

[0316] The third obtaining subunit is configured to obtain, based on the actual exposure counts of each displayable information in the third time period, and the estimated click-through rate and estimated conversion rate of each displayable information corresponding to the refined sorting model in the third time period, the sum of the second estimated click counts of each displayable information corresponding to the refined sorting model in the third time period, and the sum of the second estimated conversion counts of each displayable information in the third time period;

[0317] Based on the actual exposure counts of each displayable information in the third time period, and the estimated click-through rate and estimated conversion rate of each displayable information corresponding to the rough sorting model in the third time period, obtain the sum of the third estimated click counts of each displayable information corresponding to the rough sorting model in the third time period, and the sum of the third estimated conversion counts of each displayable information in the third time period;

[0318] Based on the sum of the second estimated click counts of each displayable information in the third time period, and the sum of the third estimated click counts of each displayable information in the third time period, obtain the initial value of the second click-through rate calibration coefficient of the rough sorting model;

[0319] Based on the sum of the second estimated conversion counts of each displayable information in the third time period, and the sum of the third estimated conversion counts of each displayable information in the third time period, obtain the initial value of the second conversion rate calibration coefficient of the rough sorting model.

[0320] Optionally, the predicted display result includes an estimated click-through rate and an estimated conversion rate; the calibration coefficient includes a click-through rate calibration coefficient and a conversion rate calibration coefficient;

[0321] The fourth obtaining subunit is configured to obtain, based on the actual exposure counts of each displayable information in the fourth time period, and the estimated click-through rate and estimated conversion rate of each displayable information corresponding to the refined sorting model in the fourth time period, the estimated click counts and estimated conversion counts of each displayable information corresponding to the refined sorting model in the fourth time period;

[0322] Based on the actual exposure counts of each displayable information in the fourth time period, and the estimated click-through rate and estimated conversion rate of each displayable information corresponding to the rough sorting model in the fourth time period, obtain the estimated click counts and estimated conversion counts of each displayable information corresponding to the rough sorting model in the fourth time period;

[0323] Based on the estimated click counts of each displayable information corresponding to the refined sorting model in the fourth time period, and the estimated click counts of each displayable information corresponding to the rough sorting model in the fourth time period, obtain the target value of the click-through rate calibration coefficient of each displayable information corresponding to the rough sorting model;

[0324] Based on the estimated conversion numbers of each displayable information in the fourth time period corresponding to the refined sorting model, and the estimated conversion numbers of each displayable information in the fourth time period corresponding to the rough sorting model, obtain the target values of the conversion rate calibration coefficients of each displayable information corresponding to the rough sorting model respectively.

[0325] Optionally, the fourth obtaining unit is configured to perform a weighted sum on the initial value of the second calibration coefficient and the target values of the calibration coefficients of each displayable information corresponding to the rough sorting model respectively, to obtain the calibration coefficients of each displayable information corresponding to the rough sorting model respectively.

[0326] In summary, the information display device provided in the embodiment of the present application can obtain the calibration coefficients of each sub-sorting model through the actual display results of each displayable information and the predicted display results of the cascade sub-sorting models of the displayable information sorting model during the information display process, so as to calibrate the displayable information sorting model. Based on the sorting results of the calibrated displayable information sorting model, each displayable information is displayed, realizing the update of the displayable information sorting model based on the prediction effect and the actual display effect of the cascade multi-level displayable information sorting model, thereby improving the accuracy of the displayable information sorting model, and further improving the display effect of each displayable information.

[0327] Figure 9 It is a structural block diagram of a computer device 900 shown according to an exemplary embodiment. The computer device can be implemented as the server in the above solution of the present application. The computer device 900 includes a central processing unit (CPU) 901, a system memory 904 including a random access memory (RAM) 902 and a read-only memory (ROM) 903, and a system bus 905 connecting the system memory 904 and the central processing unit 901. The computer device 900 further includes a basic input / output system (Input / Output system, I / O system) 906 for facilitating the transmission of information between various components within the computer, and a mass storage device 907 for storing an operating system 913, application programs 914, and other program modules 915.

[0328] The basic input / output system 906 includes a display 908 for displaying information and input devices 909 such as a mouse, keyboard, etc. for user input of information. Both the display 908 and the input devices 909 are connected to the central processing unit 901 through an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may also include an input / output controller 910 for receiving and processing inputs from a plurality of other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, printer, or other types of output devices.

[0329] The mass storage device 907 is connected to the central processing unit 901 through a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable medium provide non-volatile storage for the computer device 900. That is, the mass storage device 907 may include computer-readable media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0330] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, erasable programmable read-only registers (EPROM), electrically-erasable programmable read-only memory (EEPROM) flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cartridges, tapes, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage media is not limited to the above several types. The above system memory 904 and mass storage device 907 may be collectively referred to as memory.

[0331] According to various embodiments of the present disclosure, the computer device 900 may also be operated by a remote computer on a network through a network such as the Internet. That is, the computer device 900 may be connected to the network 912 through the network interface unit 911 connected to the system bus 905, or in other words, the network interface unit 911 may also be used to connect to other types of networks or remote computer systems (not shown).

[0332] The memory further includes at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, code set or instruction set is stored in the memory. The central processing unit 901 implements all or part of the steps in the calibration method of the displayable information sorting model shown in the above various embodiments by executing the at least one instruction, at least one program, code set or instruction set.

[0333] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are to be considered as exemplary only, and the true scope and spirit of the present application are pointed out by the following claims.

[0334] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. An information display method, characterized in that, The method includes: Obtaining the actual display results of each displayable information; the displayable information is information selected for display by predicting the prediction probability of receiving a specified user operation through a displayable information ranking model, and arranging them according to the predicted prediction probability; the displayable information ranking model includes at least two cascaded sub-ranking models, and the actual display result refers to the actual data received after the specified user operation after each displayable information is displayed on the display platform; the at least two sub-ranking models include a rough selection ranking model and a refined selection ranking model cascaded after the rough selection ranking model; Obtaining the predicted display results of each displayable information corresponding to the at least two sub-ranking models, where the predicted display results are used to indicate the prediction probabilities of the at least two sub-ranking models for receiving the specified user operation for each displayable information respectively; Based on the actual display results of each displayable information and the predicted display results of each displayable information corresponding to the refined selection ranking model respectively, obtaining the calibration coefficients of each displayable information corresponding to the refined selection ranking model; Based on the predicted display results of each displayable information corresponding to the refined selection ranking model respectively and the predicted display results of each displayable information corresponding to the rough selection ranking model respectively, obtaining the calibration coefficients of each displayable information corresponding to the rough selection ranking model; Based on the calibration coefficients of the at least two sub-ranking models, correcting the prediction probabilities of the at least two sub-ranking models for each displayable information subsequently; Calculating the cost per thousand impressions of each displayable information based on the corrected prediction probabilities of each displayable information and arranging them for displaying each displayable information.

2. The method according to claim 1, characterized in that, The step of obtaining the calibration coefficients of each displayable information corresponding to the refined selection ranking model based on the actual display results of each displayable information and the predicted display results of each displayable information corresponding to the refined selection ranking model respectively includes: Based on the actual display results of each displayable information and the predicted display results of each displayable information corresponding to the refined selection ranking model respectively, obtaining the initial value of the first calibration coefficient of the refined selection ranking model and the target values of the calibration coefficients of each displayable information corresponding to the refined selection ranking model; Based on the initial value of the first calibration coefficient and the target values of the calibration coefficients of each displayable information corresponding to the refined selection ranking model, obtaining the calibration coefficients of each displayable information corresponding to the refined selection ranking model.

3. The method according to claim 2, characterized in that, The step of obtaining the initial value of the first calibration coefficient of the refined selection ranking model and the target values of the calibration coefficients of each displayable information corresponding to the refined selection ranking model based on the actual display results of each displayable information and the predicted display results of each displayable information corresponding to the refined selection ranking model respectively includes: Obtain the initial value of the first calibration coefficient of the selected sorting model based on the actual display results of each displayable information in the first time period and the predicted display results of each displayable information corresponding to the selected sorting model in the first time period; Based on the actual display results of each displayable information in the second time period, obtain the target values of the calibration coefficients of each displayable information corresponding to the selected sorting model.

4. The method according to claim 3, wherein The actual display results include the actual number of exposures, the actual number of clicks, and the actual number of conversions; the predicted display results include the estimated click-through rate and the estimated conversion rate; the calibration coefficients include the click-through rate calibration coefficient and the conversion rate calibration coefficient; The obtaining the initial value of the first calibration coefficient of the selected sorting model based on the actual display results of each displayable information in the first time period and the predicted display results of each displayable information corresponding to the selected sorting model in the first time period includes: Obtain the sum of the actual number of clicks of each displayable information in the first time period and the sum of the actual number of conversions of each displayable information in the first time period; Based on the actual number of exposures of each displayable information in the first time period, the estimated click-through rate and the estimated conversion rate of each displayable information corresponding to the selected sorting model in the first time period, obtain the sum of the first estimated number of clicks of each displayable information corresponding to the selected sorting model in the first time period and the sum of the first estimated number of conversions of each displayable information in the first time period; Based on the sum of the actual number of clicks of each displayable information in the first time period and the sum of the first estimated number of clicks, obtain the initial value of the first click-through rate calibration coefficient of the selected sorting model; Based on the sum of the actual number of conversions of each displayable information in the first time period and the sum of the first estimated number of conversions, obtain the initial value of the first conversion rate calibration coefficient of the selected sorting model.

5. The method according to claim 3, characterized in that The actual display results include the actual number of exposures, the actual number of clicks, and the actual number of conversions; the predicted display results include the estimated click-through rate and the estimated conversion rate; the calibration coefficients include the click-through rate calibration coefficient and the conversion rate calibration coefficient; The obtaining the target values of the calibration coefficients of each displayable information corresponding to the selected sorting model based on the actual display results of each displayable information in the second time period includes: Obtain the actual number of clicks of each displayable information in the second time period and the actual number of conversions of each displayable information in the second time period; Based on the actual number of exposures of each displayable information in the second time period, the estimated click-through rate and the estimated conversion rate of each displayable information corresponding to the selected sorting model in the second time period, obtain the estimated number of clicks and the estimated number of conversions of each displayable information corresponding to the selected sorting model in the second time period; Obtain the target values of click-through rate calibration coefficients corresponding to the respective displayable information for the refined sorting model based on the actual number of clicks of the respective displayable information during the second time period and the estimated number of clicks of the respective displayable information during the second time period corresponding to the refined sorting model; Obtain the target values of conversion rate calibration coefficients corresponding to the respective displayable information for the refined sorting model based on the actual number of conversions of the respective displayable information during the second time period and the estimated number of conversions of the respective displayable information during the second time period corresponding to the refined sorting model.

6. The method according to claim 2, wherein The obtaining of the calibration coefficients corresponding to the respective displayable information for the refined sorting model based on the initial value of the first calibration coefficient and the target values of the calibration coefficients corresponding to the respective displayable information for the refined sorting model includes: Perform weighted summation on the initial value of the first calibration coefficient and the target values of the calibration coefficients corresponding to the respective displayable information for the refined sorting model to obtain the calibration coefficients corresponding to the respective displayable information for the refined sorting model.

7. The method according to claim 1, characterized in that, The obtaining of the calibration coefficients corresponding to the respective displayable information for the rough sorting model based on the predicted display results corresponding to the respective displayable information for the refined sorting model and the predicted display results corresponding to the respective displayable information for the rough sorting model includes: Based on the predicted display results corresponding to the respective displayable information for the refined sorting model and the predicted display results corresponding to the respective displayable information for the rough sorting model, obtain the initial value of the second calibration coefficient of the rough sorting model and the target values of the calibration coefficients corresponding to the respective displayable information for the rough sorting model; Based on the initial value of the second calibration coefficient and the target values of the calibration coefficients corresponding to the respective displayable information for the rough sorting model, obtain the calibration coefficients corresponding to the respective displayable information for the rough sorting model.

8. The method according to claim 7, wherein The obtaining of the initial value of the second calibration coefficient of the rough sorting model and the target values of the calibration coefficients corresponding to the respective displayable information for the rough sorting model based on the predicted display results corresponding to the respective displayable information for the refined sorting model and the predicted display results corresponding to the respective displayable information for the rough sorting model includes: Based on the predicted display results corresponding to the respective displayable information for the refined sorting model during the third time period and the predicted display results corresponding to the respective displayable information for the rough sorting model during the third time period, obtain the initial value of the second calibration coefficient of the rough sorting model; Based on the predicted display results corresponding to the respective displayable information for the refined sorting model during the fourth time period, obtain the target values of the calibration coefficients corresponding to the respective displayable information for the rough sorting model.

9. The method according to claim 8, wherein The predicted display results include estimated click-through rate and estimated conversion rate; the calibration coefficients include click-through rate calibration coefficients and conversion rate calibration coefficients; Obtaining an initial value of a second calibration coefficient of the rough sorting model based on the predicted display results of the respective displayable information corresponding to the refined sorting model in the third time period and the predicted display results of the respective displayable information corresponding to the rough sorting model in the third time period includes: Based on the actual exposure numbers of the respective displayable information in the third time period, and the estimated click-through rates and estimated conversion rates of the respective displayable information corresponding to the refined sorting model in the third time period, obtaining the sum of the second estimated click numbers of the respective displayable information corresponding to the refined sorting model in the third time period, and the sum of the second estimated conversion numbers of the respective displayable information in the third time period; Based on the actual exposure numbers of the respective displayable information in the third time period, and the estimated click-through rates and estimated conversion rates of the respective displayable information corresponding to the rough sorting model in the third time period, obtaining the sum of the third estimated click numbers of the respective displayable information corresponding to the rough sorting model in the third time period, and the sum of the third estimated conversion numbers of the respective displayable information in the third time period; Based on the sum of the second estimated click numbers of the respective displayable information in the third time period and the sum of the third estimated click numbers of the respective displayable information in the third time period, obtaining an initial value of the second click-through rate calibration coefficient of the rough sorting model; Based on the sum of the second estimated conversion numbers of the respective displayable information in the third time period and the sum of the third estimated conversion numbers of the respective displayable information in the third time period, obtaining an initial value of the second conversion rate calibration coefficient of the rough sorting model.

10. The method according to claim 8, characterized in that The predicted display results include an estimated click-through rate and an estimated conversion rate; the calibration coefficient includes a click-through rate calibration coefficient and a conversion rate calibration coefficient; Obtaining a target value of the calibration coefficient of the respective displayable information corresponding to the rough sorting model based on the predicted display results of the respective displayable information corresponding to the refined sorting model in the fourth time period includes: Based on the actual exposure numbers of the respective displayable information in the fourth time period, and the estimated click-through rates and estimated conversion rates of the respective displayable information corresponding to the refined sorting model in the fourth time period, obtaining the estimated click numbers and estimated conversion numbers of the respective displayable information corresponding to the refined sorting model in the fourth time period; Based on the actual exposure numbers of the respective displayable information in the fourth time period, and the estimated click-through rates and estimated conversion rates of the respective displayable information corresponding to the rough sorting model in the fourth time period, obtaining the estimated click numbers and estimated conversion numbers of the respective displayable information corresponding to the rough sorting model in the fourth time period; Based on the estimated click-through numbers of each displayable information during the fourth time period corresponding to the refined ranking model, and the estimated click-through numbers of each displayable information during the fourth time period corresponding to the rough ranking model, obtain the target values of the click-through rate calibration coefficients of each displayable information corresponding to the rough ranking model; Based on the estimated conversion numbers of each displayable information during the fourth time period corresponding to the refined ranking model, and the estimated conversion numbers of each displayable information during the fourth time period corresponding to the rough ranking model, obtain the target values of the conversion rate calibration coefficients of each displayable information corresponding to the rough ranking model.

11. The method according to claim 7, characterized in that, The obtaining the calibration coefficients of each displayable information corresponding to the rough ranking model based on the initial value of the second calibration coefficient and the target values of the calibration coefficients of each displayable information corresponding to the rough ranking model includes: Performing weighted summation on the initial value of the second calibration coefficient and the target values of the calibration coefficients of each displayable information corresponding to the rough ranking model to obtain the calibration coefficients of each displayable information corresponding to the rough ranking model.

12. An information display device, characterized in that, The apparatus includes: A first acquisition module, configured to acquire the actual display results of each displayable information; the displayable information is the information selected for display by predicting the probability of receiving a specified user operation through a displayable information ranking model; the displayable information ranking model includes at least two cascaded sub-ranking models, and the actual display result refers to the actual data of receiving the specified user operation after each displayable information is displayed on the display platform; the at least two sub-ranking models include a rough ranking model and a refined ranking model cascaded after the rough ranking model; A second acquisition module, configured to acquire the predicted display results of each displayable information corresponding to the at least two sub-ranking models, where the predicted display results are used to indicate the predicted probabilities of the at least two sub-ranking models receiving the specified user operation for each displayable information respectively; A third acquisition module, configured to obtain the calibration coefficients of each displayable information corresponding to the refined ranking model based on the actual display results of each displayable information and the predicted display results of each displayable information corresponding to the refined ranking model; obtain the calibration coefficients of each displayable information corresponding to the rough ranking model based on the predicted display results of each displayable information corresponding to the refined ranking model and the predicted display results of each displayable information corresponding to the rough ranking model; A display module, configured to correct the predicted probabilities of the at least two sub-ranking models for each displayable information subsequently based on the calibration coefficients of the at least two sub-ranking models; calculate the cost per thousand impressions of each displayable information based on the corrected predicted probabilities of each displayable information and perform ranking to display each displayable information.

13. A computer device, characterized in that, The computer device includes a processor and a memory. At least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the information display method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, At least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the information display method according to any one of claims 1 to 11.

Images